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EXECUTIVE  SUMMARY 


The  technology  focus  of  this  demonstration  is  a  vehicle-towed  array  of  time-domain 
electromagnetic  sensors  designed  and  operated  to  optimize  the  classification  of  buried  munitions 
from  ancillary  metallic  debris  and  clutter.  The  demonstrated  technology,  known  as  the 
TEMTADS  array,  consists  of  25  co-located  transmit  (Tx)  and  receive  (Rx)  pairs  arranged  in  a 
5x5  fixed  grid  configuration.  The  TxRx  pairs  are  separated  by  40cm  center-to-center,  which 
results  in  a  2m  square  array.  Transient  decay  data  are  collected  for  each  Rx  coil  with  a  500  kHz 
sample  rate  until  25ms  after  turn  off  of  the  excitation  pulse.  The  25  Tx  coils  are  operated 
individually  while  recording  with  all  25  Rx,  resulting  in  625  spatial  measurements.  The  system 
is  designed  to  interrogate  subsurface  sources  one  at  a  time  sequentially.  It  collects  data  while 
stationary. 

The  raw  signature  data  from  the  TEMTADS  array  reflect  details  of  the  sensor/target  geometry  as 
well  as  inherent  EMI  response  characteristics  of  the  targets  themselves.  In  order  to  separate  out 
the  intrinsic  target  response  properties  from  sensor/target  geometry  effects  the  measured 
signature  is  inverted  to  estimate  principal  axis  magnetic  polarizabilities  using  a  standard  induced 
dipole  response  model. 

The  TEMTADS  array  was  designed  to  combine  the  data  quality  advantages  of  a  gridded  survey 
with  the  coverage  efficiencies  of  a  vehicular  system.  The  design  goal  of  this  system  is  to  collect 
data  equal,  if  not  better,  in  quality  to  the  best  gridded  surveys  (the  relative  position  and 
orientation  of  the  sensors  being  known  better  than  for  gridded  data)  while  prosecuting  many 
more  targets  each  field  day. 

The  objective  of  this  demonstration  was  to  validate  the  discrimination  technology  through  two 
blind  tests.  The  first  demonstration  was  conducted  at  the  Aberdeen  Proving  Ground  (APG) 
Standardized  UXO  Test  Site.  The  second  demonstration  was  conducted  as  part  of  the  ESTCP 
UXO  Classification  Study  at  the  former  Camp  San  Luis  Obispo.  At  each  demonstration,  the  site 
had  been  blind  seeded  with  a  significant  number  of  intact,  inert  UXO  types  to  challenge  UXO 
classification  systems  and  methodologies. 

The  perfonnance  metrics  used  to  monitor  the  success  of  the  technology  relate  to  production  rate, 
accuracy  of  inverted  features,  analysis  time,  correct  classification,  and  ease  of  use.  As  detailed 
in  later  sections,  all  performance  objectives  were  successfully  met.  At  the  APG  site,  the  system 
was  able  to  interrogate  over  150  targets  per  day.  The  inverted  source  locations  were  less  than  10 
cm  in  northing  and  easting  from  the  ground  truth  locations.  The  analysis,  which  required 
roughly  15  minutes  per  target,  resulted  in  a  false  alann  reduction  by  over  50%  with  95%  correct 
identification  of  munitions. 

At  the  former  Camp  San  Luis  Obispo  site,  the  system  acquired  data  at  over  125  anomalies  per 
day.  All  data  were  checked  for  quality  on  site  and  at  pace  with  data  collection.  The  average 
error  in  location  was  less  than  5  cm  in  northing  and  easting  and  the  average  error  in  depth 
estimation  was  less  than  5  cm. 
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Qualitatively,  the  TEMTADS  array  was  easy  to  use  and  proved  to  be  a  robust  and  reliable  sensor 
platform. 


The  primary  limitation  of  this  technology  relates  to  the  size  of  the  system  and  site  access.  The 
array  is  a  2-m  square  so  fields  where  the  vegetation  or  topography  interferes  with  passage  of  a 
trailer  that  size  will  not  be  amenable  to  the  use  of  the  5x5  array. 
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1.0 


INTRODUCTION 


1.1  BACKGROUND 

Unexploded  Ordnance  (UXO)  detection  and  remediation  is  a  high  priority  triservice  requirement. 
As  the  Defense  Science  Board  recently  wrote:  “Today’s  UXO  cleanup  problem  is  massive  in 
scale  with  some  10  million  acres  of  land  involved.  Estimated  cleanup  costs  are  uncertain  but  are 
clearly  tens  of  billions  of  dollars.  This  cost  is  driven  by  the  digging  of  holes  in  which  no  UXOs 
are  present.  The  instruments  used  to  detect  UXOs  (generally  located  underground)  produce 
many  false  alarms,  -  i.e.,  detections  from  scrap  metal  or  other  foreign  or  natural  objects  -  ,  for 
every  detection  of  a  real  unexploded  munition  found.”  [1] 

There  is  general  agreement  that  the  best  solution  to  the  false  alarm  problem  involves  the  use  of 
EMI  sensors  which,  in  principle,  allow  the  extraction  of  target  shape  parameters  in  addition  to  a 
size  and  depth  estimate.  We,  and  others,  have  fielded  systems  with  either  time-domain  or 
frequency-domain  EMI  sensors  with  the  goal  of  extracting  reliable  target  shape  parameters  and, 
thus,  improving  the  discrimination  capability  of  our  surveys.  In  practice,  the  discrimination 
ability  of  these  sensors  has  been  limited  by  signal-to-noise  limitations.  Three  of  the  largest  noise 
tenns  are  inherent  sensor  noise,  motion-induced  noise,  and  sensor  location  uncertainty. 

The  three  most  successful  demonstrations  of  EMI-based  discrimination  all  involved  cued 
detection  with  gridded  collection  of  EMI  data  [2-4].  The  success  of  the  gridded  data  collections 
was  due  to  the  combination  of  minimal  location  uncertainty,  no  motion-induced  noise,  and 
sufficient  SNR.  The  downside  of  the  implementations  previously  demonstrated  is  that  they  were 
relatively  slow  and  inefficient,  especially  on  a  large  site.  We  have  constructed  an  EMI  sensor 
array  that  combines  the  classification  ability  of  a  gridded  survey  with  the  coverage  efficiency  of 
a  vehicular  array.  By  coming  to  a  stop  over  each  target  to  be  investigated  we  are  able  to  obtain 
all  the  benefits  of  a  gridded  survey  (negligible  relative  sensor  location  uncertainty,  no  motion- 
induced  noise,  and  high  SNR)  while  moving  rapidly  to  the  next  target  with  no  set-up  required 
gives  us  the  coverage  efficiency  required  for  practical  success. 

1.2  OBJECTIVE  OF  THE  DEMONSTRATION 

The  objective  of  this  demonstration  was  to  validate  the  technology  through  a  series  of  blind  test 
demonstrations.  We  conducted  a  shake-down  demonstration  of  the  technology  at  our  Blossom 
Point,  MD  field  site  but  a  blind  test  is  the  only  true  measure  of  system  perfonnance.  The  first 
demonstration  was  conducted  at  the  Aberdeen  Proving  Ground  (APG)  Standardized  UXO  Test 
Site.  The  second  demonstration  was  conducted  as  part  of  the  ESTCP  UXO  Classification  Study 
at  the  former  Camp  San  Luis  Obispo.  At  each  demonstration,  the  site  had  been  blind  seeded 
with  a  significant  number  of  intact,  inert  UXO  types  to  challenge  UXO  classification  systems 
and  methodologies.  Demonstration  scoring  was  conducted  by  IDA  to  maintain  the  integrity  of 
the  ground  truth. 


Manuscript  approved  July  29,  2010. 


1 


1.3  REGULATORY  DRIVERS 


Stakeholder  acceptance  of  the  use  of  classification  techniques  on  real  sites  will  require 
demonstration  that  these  techniques  can  be  deployed  efficiently  and  with  high  probability  of 
discrimination.  The  first  step  in  this  process  is  to  demonstrate  acceptable  performance  on 
synthetic  test  sites  such  as  that  at  Aberdeen.  As  a  second  step,  demonstration  on  a  carefully 
prepared  and  blind-seeded  live  site  presents  a  more  real-world  scenario  while  providing 
sufficiently  complete  validation  data  to  accurately  determine  system  performance.  After  these 
hurdles  have  been  passed,  successful  demonstration  at  a  live  site  will  further  facilitate  regulatory 
acceptance  of  the  UXO  classification  technology  and  methodology. 

2.0  TECHNOLOGY 

2.1  TECHNOLOGY  DESCRIPTION 

2.1.1  EMI  Sensors 

The  EMI  sensor  used  in  the  TEMTADS  array  is  based  on  the  Navy-funded  Advanced  Ordnance 
Locator  (AOL),  developed  by  G  &  G  Sciences.  The  AOL  consists  of  three  transmit  coils 
arranged  in  a  1-m  cube;  we  have  adopted  the  transmit  (Tx)  and  receive  (Rx)  subsystems  of  this 
sensor  directly,  converted  to  a  5  x  5  array  of  35  cm  sensors,  and  made  minor  modifications  to  the 
control  and  data  acquisition  computer  to  make  it  compatible  with  our  deployment  scheme. 

A  photograph  of  an  individual  sensor  element  under  construction  is  shown  in  the  left  panel  of 
Figure  2-1.  The  transmit  coil  is  wound  around  the  outer  portion  of  the  form  and  is  35  cm  on  a 
side.  The  25-cm  receive  coil  is  wound  around  the  inner  part  of  the  form  which  is  re-inserted  into 
the  outer  portion.  An  assembled  sensor  with  the  top  and  bottom  caps  used  to  locate  the  sensor  in 
the  array  is  shown  in  the  right  panel  of  Figure  2-1. 


Figure  2-1  -  Construction  details  of  an  individual  EMI  sensor  (left  panel)  and  the  assembled  sensor  with 
end  caps  attached  (right  panel). 
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Decay  data  are  collected  with  a  500  kHz  sample  rate  until  25ms  after  turn  off  of  the  excitation 
pulse.  This  results  in  a  raw  decay  of  12,500  points;  too  many  to  be  practical.  These  raw  decay 
measurements  are  grouped  into  115  logarithmically-spaced  “gates”  whose  center  times  range 
from  42  ps  to  24.35  ms  with  5%  widths  and  are  saved  to  disk.  Examples  of  the  measured 
transmit  pulse,  raw  decay,  and  gated  decay  are  shown  in  Figure  2-2. 


Figure  2-2  -  Measured  transmit  current  (on-time  upper  panel,  off-time 
second  panel),  full  measured  signal  decay  (third  panel),  and  gated  decay 
(fourth  panel)  as  discussed  in  the  text. 

2.1.2  Sensor  Array 

The  twenty-five  individual  sensors  are  arranged  in  a  5  x  5  array  as  shown  in  Figure  2-3.  The 
center-to-center  distance  is  40  cm  yielding  a  2  m  x  2  m  array.  Also  shown  in  Figure  2-3  is  the 
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position  of  the  three  GPS  antennae  that  are  used  to  determine  the  location  and  orientation  of  the 
array  for  each  cued  measurement.  A  picture  of  the  array  mounted  on  the  MTADS  EMI  sensor 
platform  is  shown  in  Figure  2-4. 


EM  Sensor 


GPS  Antenna 


Figure  2-3  -  Sketch  of  the  EMI  sensor  array  showing  the  position  of  the 
25  sensors  and  the  three  GPS  antennae. 


Figure  2-4  -  Sensor  array  mounted  on  the  MTADS  EMI  sensor  platform. 
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2.1.3  Application  of  the  Technology 


Application  of  this  technology  is  straightforward.  A  list  of  target  positions  is  developed  from  a 
survey  by  some  geophysical  instrument;  in  the  case  of  these  demonstrations,  either  the  MTADS 
magnetometer  array  at  APG  or  the  EM61  Mkll  array  at  SLO.  This  target  file,  containing  the 
target  location  and  an  optional  flag  for  additional  ‘stacking’  or  averaging,  is  transferred  to  the 
system  control  program  which  uses  the  information  from  the  three  GPS  antennae  to  guide  the 
operator  to  position  the  array  over  each  target  in  turn.  When  positioned  over  the  target,  the  data 
acquisition  computer  steps  through  the  array  sensors  sequentially,  just  as  in  the  characterization 
measurements  discussed  in  the  following  section,  and  collects  decays  from  all  twenty-five 
receive  coils  for  each  excitation.  These  data  are  then  inverted  for  target  location  and 
characteristics.  At  the  end  of  the  EMI  data  collection,  a  few  seconds  of  platform  position  and 
orientation  data  are  collected  to  be  used  to  translate  the  inverted  target  position,  which  is,  of 
course,  relative  to  the  array,  to  absolute  position  and  orientation. 

In  the  final  version  of  this  technology,  one  could  envision  the  inversion  being  performed  while 
the  operator  is  driving  the  array  to  the  next  target.  For  these  demonstrations,  we  performed  the 
inversions  off-line  so  that  we  would  have  the  ability  to  intervene  in  the  automatic  process  as 
required.  The  EMI  and  position  data  were  transferred  to  the  analyst  several  times  each  day  for 
near  real-time  analysis  at  the  demonstration  site. 

2.2  TECHNOLOGY  DEVELOPMENT 

The  individual  sensors  (consisting  of  transmit  electronics,  transmit  and  receive  coils,  pre-amp, 
and  digitizer)  were  characterized  at  G  &  G  Sciences  before  approval  was  given  for  construction 
of  the  array.  Examples  of  the  characterization  data  are  shown  in  Figure  2-5  and  Figure  2-6. 
System  stability  is  shown  in  Figure  2-5  which  plots  the  normalized  (by  measured  transmit 
current)  response  of  a  2  in  steel  ball  at  25  cm  separation  from  the  sensor.  The  data  plotted  are 
decays  1,  1001,  2001,  and  3001  in  a  continuously-triggered  series  that  began  from  a  cold  start 
and  ran  for  150  minutes. 
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Figure  2-5  -  Measured  response  from  a  2-in  steel  sphere  25  cm  from  the 
sensor.  Decays  1,  1001,  2001,  and  3001  from  a  series  that  started  from  a 
cold  start  are  plotted  along  with  the  expected  response  from  this  target. 

For  comparison  purposes,  the  expected  response  from  this  sphere  is  plotted  in  black.  As  can  be 
seen,  the  sensor  exhibits  excellent  stability  which  is  important  for  cued  deployment. 

The  second  important  characterization  test  is  sensor  response  linearity.  Since  we  collect  decay 
data  to  late  times  and  over  several  orders  of  magnitude  in  amplitude,  the  linearity  of  system 
response  is  very  important.  To  characterize  this  property  of  the  sensor,  we  constructed  a  series 
of  copper  coils  with  nominal  decay  time  constants  of  2,  4,  and  6  ms.  The  response  of  the  three 
coils  is  shown  in  Figure  2-6  which  plots  the  measured  decay  on  semi-log  axes.  After  a  transient 
at  early  times,  the  decays  exhibit  clean  exponential  behavior  with  measured  decay  times  of  1.8, 
3.3,  and  5.8  ms.  Careful  calculation  of  the  expected  decay  times  at  the  temperature  at  which  the 
tests  were  conducted  results  in  expected  values  of  1.82,  3.26,  and  5.73  ms;  the  measured  values 
are  in  excellent  agreement  with  these. 
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Figure  2-6  -  Measured  response  from  three  calibration  coils  and  the 
background  response  between  measurements  plotted  on  a  semi-log  plot 
to  emphasize  the  exponential  nature  of  the  decay.  The  decay  time 
constants  extracted  from  the  measurements  are  listed  in  the  legend. 

After  assembly  of  the  array,  a  number  of  array  calibration  measurements  were  performed.  The 
first  task  was  to  ensure  that  each  of  the  individual  sensors  has  equivalent  response.  A  jig  was 
constructed  that  allowed  us  to  mount  a  2-in  steel  sphere  30  cm  below  each  array  element  in  turn. 
Data  collected  using  this  jig  are  shown  in  Figure  2-7.  As  can  be  seen,  the  measured  decays  from 
each  of  the  sensors  plotted  are  indistinguishable. 

The  assembled  array  was  then  used  to  measure  the  response  of  a  number  of  inert  ordnance  items 
and  stimulants  mounted  on  a  test  stand,  placed  in  our  test  pit,  or  buried  in  our  test  field.  For  each 
series  of  measurements  with  the  full  array,  we  cycle  through  the  sensors  transmitting  from  each 
in  turn.  After  each  excitation  pulse,  we  record  the  response  of  all  twenty-five  receive  coils. 
Thus,  there  are  625  (25  x  25)  individual  transmit/receive  pairs  recorded,  making  it  difficult  to 
present  a  full  measurement  in  any  coherent  way.  In  Figure  2-8,  we  plot  nine  of  the 
transmit/receive  pairs  resulting  from  excitation  of  a  40-mm  projectile  located  under  the  center  of 
the  array.  The  decays  plotted  correspond  to  the  signal  received  on  the  nine  central  sensors 
(reference  Figure  2-3  for  the  sensor  numbering)  when  that  sensor  transmits.  In  other  words,  the 
results  of  nine  individual  monostatic  measurements  are  presented. 

All  625  measurements  are  used  for  the  inversion  to  recover  target  parameters.  The  inversion 
results  for  the  decay  data  shown  in  Figure  2-8  are  shown  in  Figure  2-9.  As  we  expect  for  an 
object  with  axial  symmetry  such  as  a  40-mm  projectile,  we  recover  one  large  response 
coefficient  and  two  equal,  but  smaller  ones.  These  response  coefficients  are  the  basis  of  the 
discrimination  decisions  in  this  demonstration.  Derived  Ps  for  “Cylinder  E”  (3"  x  12"  steel 
cylinder)  in  the  test  field  are  shown  for  comparison  in  Figure  2-10. 
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Figure  2-7  -  Comparison  of  the  response  of  the  array  members.  The 
measured  decay  from  a  2-in  steel  sphere  held  30  cm  below  each  sensor  in 
turn  is  plotted.  The  decays  are  indistinguishable. 
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Figure  2-8  -  The  response  of  nine  of  the  individual  sensors  to  a  40-mm  projectile  located 
under  the  center  of  the  array. 
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Figure  2-9  -  Derived  response  coefficients  for  a  40-mm  projectile  using 
the  measurements  of  which  the  decays  shown  in  Figure  2-8  are  a  subset. 


Figure  2-10  -  Derived  response  coefficients  from  a  cued  measurement 
over  "Cylinder  E"  in  the  test  field. 

The  final  array  characterization  test  was  to  confirm  that  the  response  coefficients  we  recover  are 
invariant  to  object  position  and  orientation  under  the  array.  Figure  2-11  shows  the  derived  Ps 
plotted  for  a  4.2-in  mortar  baseplate  after  measurements  at  three  position/orientation  pairs.  As 
can  be  seen,  the  inversion  results  are  robust  to  variation  in  the  object’s  position  and  orientation. 
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Figure  2-11  -  Three  sets  of  [Is  derived  from  three  measurements  over  a 
4.2-in  mortar  baseplate  at  different  position/orientation  pairs. 

2.3  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  TEMTADS  array  is  designed  to  combine  the  data  quality  advantages  of  a  gridded  survey 
with  the  coverage  efficiencies  of  a  vehicular  system.  The  design  goal  of  this  system  is  to  collect 
data  equal,  if  not  better,  in  quality  to  the  best  gridded  surveys  (the  relative  position  and 
orientation  of  the  sensors  being  known  better  than  for  gridded  data)  while  prosecuting  many 
more  targets  each  field  day. 

There  are  obvious  limitations  to  the  use  of  this  technology.  The  array  is  a  2-m  square  so  fields 
where  the  vegetation  or  topography  interferes  with  passage  of  a  trailer  that  size  will  not  be 
amenable  to  the  use  of  the  present  array.  The  other  serious  limitation  will  be  anomaly  density. 
For  all  sensors,  there  is  a  limiting  anomaly  density  above  which  the  response  of  individual 
targets  cannot  be  separated.  We  have  chosen  relatively  small  sensors  for  this  array  which  should 
help  with  this  problem  but  we  cannot  eliminate  it.  Based  on  experiments  at  our  test  pit  at 
Blossom  Point,  the  results  of  this  demonstration,  and  work  done  on  the  former  Camp  Sibert  data 
sets,  anomaly  densities  of  300  anomalies/acre  or  higher  would  limit  the  applicability  of  this 
system  as  more  than  20%  of  the  anomalies  would  have  another  anomaly  within  a  meter.  For  low 
SNR  targets,  our  standard  data  acquisition  parameters  may  not  be  sufficient.  The  system 
software  has  built  in  the  capability  to  vary  the  data  acquisition  parameters  on-the-fly  based  on 
flags  in  the  target  file  and  can  be  reconfigured  manually  as  required.  One  area  in  need  of 
development  is  a  robust,  consolidated  data  collection  /  data  QC  methodology  for  determining 
when  there  is  a  low  SNR  anomaly  and  when  there  is  no  anomaly  present.  This  issue  is  an  area  of 
ongoing  research. 
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3.0  PERFORMANCE  OBJECTIVES 


The  TEMTADS  array  was  deployed  to  demonstrations  at  both  the  APG  Standardized  UXO  Test 
Site  and  the  ESTCP  UXO  Classification  Study  site  at  the  former  Camp  SLO.  The  performance 
objectives  for  these  demonstrations  are  summarized  in  Table  3-1.  Since  the  TEMTADS  array  is 
a  discrimination  technology,  the  performance  objectives  focus  on  the  second  step  of  the  UXO 
survey  problem;  we  assume  that  the  anomalies  from  all  targets  of  interest  have  been  detected  and 
included  on  the  target  list  that  we  worked  from. 

Due  to  the  nature  of  the  demonstrations  at  each  site  and  the  employed  scoring  methods,  separate 
Performance  Objectives  are  provided  for  each  demonstration  in  the  following  subsections. 

3.1  APG  STANDARDIZED  UXO  TEST  SITE 

Table  3-1  -  Performance  Objectives  for  the  APG  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Results 

Reduction  of 
False  Alarms 

Number  of  false 
alarms  eliminated  at 
demonstrator 
operating  point. 

•  Prioritized  dig 
list 

•  Scoring  report 
from  APG 

Reduction  of 
false  alarms  by  > 
50%  with  95% 

correct 

identification  of 
munitions 

Yes 

Location 

Accuracy 

Average  error  and 
standard  deviation  in 
both  axes  for 
interrogated  items 

•  Estimated 
location  from 
analysis 

•  Scoring  report 
from  APG 

AN  and  AE<  10 

cm 

ctN  and  ctE  <  1 5 

cm 

Yes 

Production 

Rate 

Number  of  targets 
interrogated  each  day 

•  Log  of  field 
work 

75  targets  per 
day 

Yes 

Analysis 

Time 

Average  time 
required  for 
inversion  and 
classification 

•  Log  of  analysis 
work 

1 5  min  per  target 

Yes 

Ease  of  Use 

•  Feedback  from 
operator  on  ease 
of  use 

Operator  comes 
to  work  smiling 

Yes 
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3.1.1  OBJECTIVE:  REDUCTION  OF  FALSE  ALARMS 


This  is  the  primary  measure  of  the  effectiveness  of  this  technology.  By  collecting  high-quality, 
precisely-located  data,  we  expect  to  be  able  to  discriminate  munitions  from  scrap  and  frag  with 
high  efficiency. 

3. 1.1.1.  Metric 

At  a  seeded  test  site  such  as  the  APG  standardized  test  site,  the  metric  for  false  alarm  elimination 
is  straightforward.  We  prepared  a  ranked  dig  list  for  the  targets  we  interrogated  with  a  dig/no¬ 
dig  threshold  indicated  and  ATC  personnel  used  their  automated  scoring  algorithms  to  assess  our 
results. 

3.1.1.2. Data  Requirements 

The  identification  of  most  of  the  items  in  the  test  field  is  known  to  the  test  site  operators.  Our 
ranked  dig  list  is  the  input  for  this  standard  and  ATC’s  standard  scoring  is  the  output. 

3. 1.1.3. Success  Criteria 

The  objective  will  be  considered  to  be  met  if  more  than  50%  of  the  non-munitions  items  were 
labeled  as  no-dig  while  retaining  95%  of  the  munitions  items  on  the  dig  list. 

3.1.1.4.Results 

This  Objective  was  successfully  met.  The  TEMTADS  surveyed  anomalies  detected  by  the 
MTADS  magnetometer  system  in  the  Blind  Grid  and  Indirect  Fire  Areas.  Efficiency  (E)  and 
false  positive  rejection  rate  (Rfp)  are  used  to  score  discrimination  performance  ability  at  two 
specific  operating  points  on  a  ROC  curve:  one  at  the  point  where  no  decrease  in  Pd  is  incurred 
and  the  other  at  the  operator-selected  threshold.  Efficiency  is  defined  as  the  fraction  of  detected 
ordnance  correctly  classified  as  ordnance  and  the  false  positive  rejection  rate  is  defined  as  the 
fraction  of  detected  clutter  correctly  classified  as  clutter.  The  results  for  the  Blind  Grid  and 
Indirect  Fire  Test  Areas  were  E=  0.99  and  Rfp  =  0.99,  and  E=  0.98  and  Rfp  =  0.92  at  the 
operating  point,  respectively.  With  no  loss  of  Pd,  the  results  were  E=  1.00  and  Rfp  =  0.95,  and 
E=  1.00  and  Rfp  =  0.58,  for  the  Blind  Grid  and  the  Indirect  Fire  Areas  respectively.  These  data 
are  summarized  from  Tables  7a  and  7c  of  Reference  15. 

3.1.2  OBJECTIVE:  LOCATION  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  location  of  the  targets  marked  to  be  dug.  Large  location  errors  lead  to  confusion 
among  the  UXO  techs  assigned  to  the  remediation  costing  time  and  often  leading  to  removal  of  a 
small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 
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3.1.2.  l.Metric 


As  above,  the  metric  for  location  accuracy  is  straightforward  at  a  seeded  test  site  such  as  the 
APG  standardized  test  site.  We  provided  an  estimated  position  for  all  targets  we  interrogated 
and  ATC  personnel  used  their  automated  scoring  algorithms  to  assess  our  results. 

3.1.2.2.Data  Requirements 

The  location  of  most  of  the  items  in  the  test  field  is  known  to  the  test  site  operators.  Our  dig  list 
is  the  input  for  this  standard  and  ATC’s  standard  scoring  is  the  output. 

3. 1.2.3. Success  Criteria 

The  objective  will  be  considered  to  be  met  if  the  average  position  error  was  less  than  10  cm  in 
both  dimensions  (low  bias)  and  the  standard  deviation  of  each  dimension  was  less  than  15  cm 
(accurate  location). 

3.1.2.4.Results 

This  Objective  was  successfully  met.  The  location  accuracy  of  fit  parameters  generated  from  the 
TEMTADS  array  data,  taken  from  Tables  9a  and  9c  of  Reference  15,  are  within  5  cm 
horizontally  (la)  and  6  cm  vertically  (la)  for  the  Indirect  Fire  Area.  Horizontal  errors  are  not 
calculated  for  the  Blind  Grid.  The  vertical  accuracy  was  4  cm  (la)  for  the  Blind  Grid. 

3.1.3  OBJECTIVE:  PRODUCTION  RATE 

Even  if  the  performance  of  the  technology  on  the  two  metrics  above  is  satisfactory,  there  is  an 
economic  metric  to  consider.  There  is  a  known  cost  of  remediating  a  suspected  munitions  item. 
If  the  cost  to  interrogate  a  target  is  greater  than  this  cost,  the  technology  will  be  useful  only  at 
sites  with  special  conditions  or  target  values.  Note,  however,  that  in  its  ultimate  implementation 
this  technology  will  result  in  reacquisition,  cued  interrogation,  and  target  flagging  in  one  visit  to 
the  site. 

3.1.3. 1. Metric 

The  number  of  targets  interrogated  per  day  is  the  metric  for  this  objective.  Combined  with  the 
daily  operating  cost  of  the  technology  this  gives  the  per-item  cost. 

3.1.3.2. Data  Requirements 

Survey  productivity  was  detennined  from  a  review  of  the  ATC  demonstration  field  logs. 

3. 1.3.3. Success  Criteria 

For  this  first  demonstration,  the  objective  will  be  considered  to  be  met  if  at  least  75  targets  were 
interrogated  each  survey  day. 
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3.1.3.4.Results 


This  Objective  was  successfully  met.  Data  collection  times  are  taken  from  Table  5  of  Reference 
15.  214  anomalies  were  investigated  on  the  Blind  Grid  over  the  course  of  1.43  work  days,  or  on 
average  149  anomalies  /  work  day.  694  anomalies  were  investigated  in  the  Indirect  Fire  Area 
over  the  course  of  32  hours  and  30  minutes,  or  on  average  170  anomalies  /  work  day. 

3.1.4  OBJECTIVE:  ANALYSIS  TIME 

The  ultimate  implementation  of  this  technology  will  involve  on-the-fly  analysis  and 
classification.  The  time  for  this  will  be  limited  to  the  driving  time  to  the  next  anomaly  on  the 
list.  We  will  track  the  near-real-time  analysis  time  in  this  demonstration. 

3. 1.4.1.  Metric 

The  time  required  for  inversion  and  classification  per  anomaly  is  the  metric  for  this  objective 

3.1.4.2. Data  Requirements 

Analysis  time  was  determined  from  a  review  of  the  data  analysis  logs. 

3. 1.4.3. Success  Criteria 

For  this  first  demonstration,  the  objective  will  be  considered  to  be  met  if  the  average  inversion 
and  classification  time  was  less  than  15  min. 

3.1.4.4.Results 

This  Objective  was  successfully  met.  The  average  inversion  time  per  target  was  approximately 

2.5  minutes  on  our  field  laptop  computer.  Including  this,  the  average  analysis  time  amounted  to 

12.5  minutes  per  anomaly.  Since  this  was  the  first  extensive  test  of  the  system  in  field  mode,  we 
took  the  opportunity  to  consider  various  discrimination  and  classification  methods,  some  of 
which  proved  unfruitful.  As  a  result  of  lessons  learned  from  this  undertaking,  we  expect  the 
average  analysis  time  for  future  field  runs  to  be  less  than  that  obtained  here. 

3.1.5  OBJECTIVE:  EASE  OF  USE 

This  qualitative  objective  is  intended  as  a  measure  of  the  long-tenn  usability  of  the  technology. 
If  the  operator  does  not  report  that  the  technology  is  easy  to  use,  shortcuts  that  can  compromise 
the  efficiency  of  the  technology  will  begin  to  creep  into  daily  operations. 

3.1.5.1.Data  Requirements 

This  objective  was  evaluated  based  on  operator  feedback. 
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3.1.5.2.Results 


This  Objective  was  successfully  met.  Based  on  vehicle  operator  feedback,  there  were  no 
significant  limitations  to  the  efficient  use  of  the  system  in  the  field.  Several  suggestions  were 
made  for  additional  improvements  to  the  navigation  and  data  collection  software.  They  have 
been  subsequently  incorporated.  Based  on  the  suggestions,  anomalies  for  future  demonstrations 
will  be  ordered  along  lm  swaths  for  maximum  data  collection  efficiency. 
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3.2  FORMER  CAMP  SAN  LUIS  OBISPO 


Table  3-2  -  Performance  Objectives  for  the  former  Camp  SLO  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Results 

Quantitative  Performance  Objectives 

Site  Coverage 

Fraction  of  assigned 

anomalies 

interrogated 

Survey  results 

1 00%  as  allowed  for  by 
topography  /  vegetation 

Yes 

Calibration  Strip 
Results 

System  response 
consistently  matches 
physics-based  model 

System  response 
curves 

Daily  calibration 
strip  data 

<15%  rms  variation  in 
amplitude 

Down-track  location 
±  25cm 

All  response  values  fall 
within  bounding  curves 

No 

Location  Accuracy 

Average  error  and 
standard  deviation  in 
both  axes  for 
interrogated  items 

Estimated  location 
from  analyses 

Ground  truth  from 
validation  effort 

AN  and  AE  <  5  cm 

ctN  and  ctE  <10  cm 

No 

Depth  Accuracy 

Standard  deviation  in 
depth  for  interrogated 
items 

Estimated  location 
from  analyses 

Ground  truth  from 
validation  effort 

ADepth  <  5  cm 

aDepth  <10  cm 

No 

Production  Rate 

Number  of  anomalies 
investigated  each  day 

Survey  results 

Log  of  field  work 

125  anomalies/day 

Yes 

Data  Throughput 

Throughput  of  data 

QC  process 

Log  of  analysis 
work 

All  data  QC’ed  on  site 
and  at  pace  with  survey 

Yes 

Qualitative  Performance  Objective 

Reliability  and 
Robustness 

General  Observations 

Team  feedback  and 
recording  of 
emergent  problems 

Field  team  comes  to 
work  smiling 

Yes 

3.2.1  OBJECTIVE:  SITE  COVERAGE 

A  list  of  previously  identified  anomalies  will  be  provided  by  the  Program  Office.  The 
expectation  is  to  gather  cued  data  with  the  TEMTADS  system  over  each  anomaly. 
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3.2.1. l.Metric 


Site  coverage  is  defined  as  the  fraction  of  the  assigned  anomalies  surveyed  by  the  TEMTADS. 
Exceptions  are  to  be  made  for  topology  /  vegetation  interferences. 

3.2.1.2.Data  Requirements 

The  collected  data  will  be  compared  to  the  original  anomaly  list.  Any  interferences  will  be  noted 
in  the  field  log  book  as  they  occur. 

3.2. 1.3. Success  Criteria 

The  objective  will  be  considered  met  if  100%  of  the  assigned  anomalies  are  surveyed  with  the 
exception  of  areas  that  can  not  be  surveyed  due  to  topology  /  vegetation  interferences. 

3.2.1.4.Results 

This  objective  was  successfully  met.  Of  the  list  provided  by  the  Program  Office,  all  but  5  were 
measured.  Failure  to  measure  these  5  anomalies  was  due  to  the  presence  of  rocks  which 
prevented  the  operator  from  positioning  the  TEMTADS  over  the  target. 

3.2.2  OBJECTIVE:  CALIBRATION  STRIP  RESULTS 

This  objective  supports  that  each  sensor  system  is  in  good  working  order  and  collecting 
physically  valid  data  each  day.  The  calibration  strip  is  to  be  surveyed  twice  daily.  The  peak 
positive  response  of  each  emplaced  item  from  each  run  is  compared  to  the  physics-based 
response  curves  generated  prior  to  data  collection  on  site  using  each  item  of  interest. 

3.2.2. 1. Metric 

The  reproducibility  of  the  measured  response  of  each  sensor  system  to  the  items  of  interest  and 
the  comparison  of  the  response  to  the  response  predicted  by  the  physics-based  model  defines  this 
metric. 

3.2.2.2. Data  Requirements 

Response  curves  for  each  sensor  /  item  of  interest  pair  are  used  to  document  what  the  physics- 
based  response  of  the  system  to  the  item  should  be.  The  tabulated  peak  response  values  from 
each  survey  of  the  Calibration  Strip  demonstrates  the  reproducibility  and  validity  of  the  sensor 
readings. 

3.2.2.3. Success  Criteria 

The  objective  will  be  considered  met  if  all  measured  responses  fall  within  the  range  of  physically 
possible  values  based  on  the  appropriate  response  curve.  Additionally,  the  RMS  variation  in 
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responses  should  be  less  than  15%  of  the  measured  response  and  the  down-track  location  of  the 
anomaly  should  be  within  25  cm  of  the  corresponding  seeded  item’s  true  location. 

3.2.2.4. Results 

This  objective  was  not  successfully  met  in  full.  The  measured  peak  signals  for  all  of  the 
emplaced  items  are  shown  in  Figure  7-1  thru  Figure  7-4.  The  maximum  (red)  and  minimum 
(blue)  response  curves  are  plotted  for  all  objects  except  the  shotput,  which  has  only  one  curve 
due  to  symmetry.  The  two  curves  for  the  81 -mm  and  4.2-in  mortar  are  nearly  equal.  For  self- 
consistency,  we  have  plotted  the  measurements  at  the  mean  inverted  depth,  rather  than  the 
reported  depths.  The  measured  values  generally  fit  well  within  the  bounding  curves,  with  the  4.2- 
in  mortar  and  shotput  results  the  poorest,  with  a  tendency  to  underestimate  the  peak  value. 

Careful  examination  of  the  data  shows  that  this  variation  is  the  result  of  the  shot-to-shot 
precision  with  which  the  array  can  be  positioned  in  exactly  the  same  spot  each  time.  Because  the 
response  curves  are  generated  assuming  the  target  is  directly  below  the  sensor,  any  offset  in  the 
sensor  position  will  result  in  the  derived  peak  signal  being  smaller  than  that  predicted  by  the 
curve,  as  is  observed.  For  future  demonstrations,  the  metric  of  polarizability  amplitude  is 
recommended  as  these  values  are  invariant  to  array  position. 

3.2.3  OBJECTIVE:  LOCATION  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  location  of  the  targets  marked  to  be  dug.  Large  location  errors  lead  to  confusion 
among  the  UXO  technicians  assigned  to  the  remediation  costing  time  and  often  leading  to 
removal  of  a  small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

3.2.3. 1. Metric 

The  average  error  and  standard  deviation  in  both  horizontal  axes  will  be  computed  for  the  items 
which  are  selected  for  excavation  during  the  validation  phase  of  the  study. 

3.2.3.2. Data  Requirements 

The  anomaly  fit  parameters  and  the  ground  truth  for  the  excavated  items  will  be  required  to 
determine  the  performance  of  the  fitting  routines  in  terms  of  the  location  accuracy. 

3.2.3.3. Success  Criteria 

This  objective  will  be  considered  as  met  if  the  average  error  in  position  for  both  Easting  and 
Northing  quantities  is  less  than  5  cm  and  the  standard  deviation  for  both  is  less  than  10  cm. 

3.2.3.4. Results 

This  objective  was  not  successfully  met  in  full.  The  average  Northing  position  error  for  all 
measured  data  was  1.5cm,  while  the  average  Easting  position  error  was  3cm.  The  standard 
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deviations,  however,  were  larger  than  desired,  with  both  values  about  25cm.  Excluding  those 
anomalies  for  which  multiple  targets  were  found  produces  negligible  improvement.  We  suspect 
the  higher  values  are  due  to  the  large  number  of  small,  low  SNR  clutter  items,  which  result  in 
greater  uncertainty  in  both  the  measured  and  fitted  values.  Indeed,  if  we  restrict  ourselves  to 
those  anomalies  which  we  classified  as  likely  UXO,  the  Northing  and  Easting  standard 
deviations  drop  to  7cm  and  5cm,  respectively. 

3.2.4  OBJECTIVE:  DEPTH  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  depth  of  the  targets  marked  to  be  dug.  Large  depth  errors  lead  to  confusion  among 
the  UXO  technicians  assigned  to  the  remediation  costing  time  and  often  leading  to  removal  of  a 
small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

3.2.4.1. Metric 

The  standard  deviation  of  the  predicted  depths  with  respect  to  the  ground  truth  will  be  computed 
for  the  items  which  are  selected  for  excavation  during  the  validation  phase  of  the  study. 

3.2.4.2. Data  Requirements 

The  anomaly  fit  parameters  and  the  ground  truth  for  the  excavated  items  will  be  required  to 
determine  the  performance  of  the  fitting  routines  in  terms  of  the  predicted  depth  accuracy. 

3.2.4.3. Success  Criteria 

This  objective  will  be  considered  as  met  if  the  average  error  in  depth  is  less  than  5  cm  and  the 
standard  deviation  is  less  than  10  cm. 

3.2.4.4.Results 

This  objective  was  not  met  successfully  in  full.  The  average  depth  error  for  all  measured  data 
was  2.5cm.  The  standard  deviation  was  a  bit  larger  than  desired,  with  a  value  of  14cm. 
Excluding  those  anomalies  for  which  multiple  targets  were  found  and  restricting  the  analysis  to 
those  anomalies  classified  as  ‘likely  UXO’  to  exclude  low  SNR  clutter  items,  the  standard 
deviation  drops  to  7cm. 

3.2.5  OBJECTIVE:  PRODUCTION  RATE 

This  objective  considers  a  major  cost  driver  for  the  collection  of  high-density,  high-quality 
geophysical  data,  the  production  rate.  The  faster  quality  data  can  be  collected,  the  higher  the 
financial  return  on  the  data  collection  effort. 
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3.2.5.1.Metric 


The  number  of  anomalies  investigated  per  day  determines  the  production  rate  for  a  cued  survey 
system. 

3.2.5.2.Data  Requirements 

The  metric  can  be  detennined  from  the  combination  of  the  field  logs  and  the  survey  results.  The 
field  logs  record  the  amount  of  time  per  day  spent  acquiring  the  data  and  the  survey  results 
determine  the  number  of  anomalies  investigated  in  that  time  period. 

3.2. 5.3. Success  Criteria 

This  objective  will  be  considered  met  if  average  production  rate  is  at  least  125  anomalies  /  day. 

3.2.5.4.Results 

This  objective  was  successfully  met.  A  total  of  1547  anomalies  (including  redo’s)  were  measured 
over  a  10-day  run  for  an  average  of  155  anomalies/day.  The  only  days  for  which  our  average  fell 
below  our  goal  of  125  anomalies/day  were  the  final  day,  due  to  finishing  up  all  targets,  and  2 
days  in  which  necessary  vehicle  repairs  shortened  our  workday. 

3.2.6  OBJECTIVE:  DATA  THROUGHPUT 

The  collection  of  a  complete,  high-quality  data  set  with  the  sensor  platform  is  critical  to  the 
downstream  success  of  the  UXO  Classification  Study.  This  objective  considers  one  of  the  key 
data  quality  issues,  the  ability  of  the  data  analysis  workflow  to  support  the  data  collection  effort 
in  a  timely  fashion.  To  maximize  the  efficient  collection  of  high  quality  data,  a  series  of 
MTADS  standard  data  quality  check  are  conducted  during  and  immediately  after  data  collection 
on  site.  Data  which  pass  the  QC  screen  are  then  processed  into  archival  data  stores.  Individual 
anomaly  analyses  are  then  conducted  on  those  archival  data  stores.  The  data  QC  /  preprocessing 
portion  of  the  workflow  needs  to  keep  pace  with  the  data  collection  effort  for  best  performance. 

3.2.6.1. Metric 

The  throughput  of  the  data  quality  control  workflow  is  at  least  as  fast  the  data  collection  process, 
providing  real  time  feedback  to  the  data  collection  team  of  any  issues. 

3.2.6.2. Data  Requirements 

The  data  analysts  log  books  will  provide  the  necessary  data  for  determining  the  success  of  this 
metric. 
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3.2. 6.3. Success  Criteria 


This  objective  will  be  considered  met  if  all  collected  data  can  be  processed  through  the  data 
quality  control  portion  of  the  workflow  in  a  timely  fashion. 

3.2.6.4.Results 

This  objective  was  successfully  met.  Data  were  normally  downloaded  several  times  during  each 
workday,  and  quality  control  on  these  datasets  was  usually  completed  on  the  same  day.  Quality 
control  checks  successfully  caught  missed  anomalies,  a  small  number  of  corrupt  data  files,  and 
targets  which  needed  re-measuring. 

For  low  SNR  targets,  our  standard  data  acquisition  parameters  may  not  be  sufficient.  The  system 
software  has  built  in  the  capability  to  vary  the  data  acquisition  parameters  on-the-fly  based  on 
flags  in  the  target  file  and  can  be  reconfigured  manual  as  required.  To  date  these  capabilities 
have  not  been  demonstrated  and  could  potentially  have  an  impact  on  data  throughput.  A  robust, 
consolidated  data  collection  /  data  QC  methodology  for  determining  when  there  is  a  low  SNR 
anomaly  and  when  there  is  no  anomaly  present  is  necessary  to  accurately  and  efficiently  utilize 
these  capabilities.  This  issue  is  an  area  of  ongoing  research. 

3.2.7  OBJECTIVE:  RELIABILITY  AND  ROBUSTNESS 

This  objective  represents  an  opportunity  for  all  parties  involved  in  the  data  collection  process, 
especially  the  vehicle  operator,  to  provide  feedback  on  areas  where  the  process  could  be 
improved. 

3.2.7.1. Data  Requirements 

Discussions  with  the  entire  field  team  and  other  observations  will  be  used. 

3.2.7.2. Results 

This  objective  was  successfully  met.  Based  on  vehicle  operator  feedback,  there  were  no 
significant  limitations  to  the  efficient  use  of  the  system  in  the  field. 

4.0  SITE  DESCRIPTION 

Two  demonstrations  were  conducted  for  this  project.  The  first  was  conducted  at  the  APG 
Standardized  UXO  Technology  Demonstration  Site  located  at  the  Aberdeen  Proving  Ground, 
MD  during  the  period  of  May  through  June,  2008.  The  second  was  conducted  at  the  former 
Camp  San  Luis  Obispo,  CA  ESTCP  UXO  Classification  Study  Demonstration  Site  in  June, 
2009. 
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4.1  APG  STANDARDIZED  UXO  TECHNOLOGY  DEMONSTRATION  SITE 

4.1.1  SITE  SELECTION 

This  was  our  first  field  demonstration  of  this  combination  of  EMI  sensors  and  survey  mode.  As 
such,  the  demonstration  was  conducted  on  the  Standardized  UXO  Test  Site  at  APG.  The  APG 
site  is  located  close  to  our  base  of  operations  in  southern  Maryland  and  therefore  minimizes  the 
logistics  costs  of  the  deployment.  Use  of  this  site  allows  us  to  receive  validation  results  from 
near-real-world  conditions  without  incurring  the  logistics  and  intrusive  investigation  expenses 
that  would  be  required  for  a  demonstration  at  a  live  site. 

4.1.2  SITE  HISTORY 

The  Standardized  UXO  Technology  Demonstration  Site  is  adjacent  to  the  Trench  Warfare 
facility  at  the  Aberdeen  Proving  Ground.  The  specific  area  was  used  for  a  variety  of  ordnance 
tests  over  the  years.  Initial  magnetometer  and  EMI  surveys  conducted  by  the  MTADS  team 
performed  after  a  “mag  and  flag”  survey  of  the  same  area  identified  over  a  thousand  remaining 
anomalies.  These  data  were  used  for  a  final  clean  up  of  the  site  prior  to  the  emplacement  of  the 
original  test  items.  Prior  to  the  two  subsequent  reconfiguration  events,  unexplained  anomalies 
identified  by  demonstrators  using  the  site  were  also  investigated  and  removed. 

4.1.3  SITE  TOPOGRAPHY  AND  GEOLOGY 

According  to  the  soils  survey  conducted  for  the  entire  area  of  APG  in  1998,  the  test  site  consists 
primarily  of  Elkton  Series  type  soil  [5].  The  Elkton  Series  consist  of  very  deep,  slowly 
penneable,  poorly  drained  soils.  These  soils  formed  in  silty  aeolin  sediments  and  the  underlying 
loamy  alluvial  and  marine  sediments.  They  are  on  upland  and  lowland  flats  and  in  depressions 
of  the  Mid-Atlantic  Coastal  Plain.  Slopes  range  from  0  to  2  percent. 

Overall,  the  demonstration  site  is  relatively  flat  and  level.  There  are  some  low-lying  areas  in  the 
northwest  portion  of  the  site  that  tend  to  have  standing  water  during  the  wet  periods  of  the  year. 
The  current  sensor  system  is  not  sufficiently  weatherproofed  to  operate  through  standing  water. 
However,  during  the  most  recent  reconfiguration,  the  areas  most  prone  to  being  underwater  were 
excluded  from  the  survey  scenarios.  Anomalies  that  were  located  underwater  or  nearby  to  water 
at  the  time  of  survey  were  deferred  until  the  end  of  the  survey  and  were  interrogated  by  carefully, 
if  less  efficiently,  maneuvering  the  array  into  position. 

4.1.4  MUNITIONS  CONTAMINATION 

The  area  currently  occupied  by  the  Standardized  site  has  seen  an  extensive  history  of  munitions 
use.  As  an  example,  in  2003  we  conducted  a  magnetometer  survey  of  a  previously  unremediated 
area  directly  adjacent  to  the  Standardized  site  [6],  In  a  survey  area  of  approximately  1  hectare, 
we  identified  2479  anomalies,  of  which  1921  were  amenable  to  a  model  fit  using  our  standard 
analysis.  Historical  records  provided  by  ATC  and  previous  remediation  results  indicated  that  the 
likely  munitions  of  interest  for  this  site  were: 
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•  Grenades,  Mkl,  Mkll,  and  French  VB  Rifle  w/o  chute 

•  Grenades,  French  VB  Rifle  w/  chute 

•  60mm  mortars  (including  2”  Smoke) 

•  3”  Stokes  (Smoke  and  HE) 

•  105  mm  projectiles 

•  155  mm  projectiles 

4,1.5  SITE  GEODETIC  CONTROL  INFORMATION 

There  are  two  first-order  points  on  the  site  for  use  as  GPS  base  station  points.  Their  reported 
coordinates  are  listed  in  Table  4-1.  The  horizontal  datum  for  all  values  is  NAD83.  The  vertical 
control  is  referenced  to  the  NAVD88  datum  and  the  Geoid03  geoid.  Point  477  was  used  as  the 
GPS  base  station  point  for  the  entirety  of  this  demonstration. 

Table  4-1  -  Geodetic  Control  at  the  APG  Demonstration  Site 


ID 

Latitude 

Longitude 

Elevation 

Northing 

Easting 

HAE 

477 

39°  28'  18.63880"  N 

76°  07'  47.71815"W 

10.669  m 

4,369,749.013 

402,810.038 

-22.545 

478 

39°  28'  04.24219"  N 

76°  07'  48.50439"W 

11.747  m 

4,369,305.416 

402,785.686 

-21.473 

4.1.6  SITE  CONFIGURATION 

Figure  4-1  is  a  map  of  the  Standardized  UXO  Technology  Demonstration  Site  at  APG.  The 
Calibration  and  Blind  Grids  are  shown  along  with  the  various  Open  Field  Areas. 
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Figure  4-1  -  Map  of  the  reconfigured  APG  Standardized  UXO  Test  Site. 
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4.2  FORMER  CAMP  SAN  LUIS  OBISPO 


The  site  description  material  reproduced  here  is  taken  from  the  recent  SI  report  [7],  More  details 
can  be  obtained  in  the  report.  The  former  Camp  SLO  is  approximately  2,101  acres  situated  along 
Highway  1,  approximately  five  miles  northwest  of  San  Luis  Obispo,  California.  The  majority  of 
the  area  consists  of  mountains  and  canyons.  The  site  for  this  demonstration  is  a  mortar  target  on 
hilltop  in  MRS  05  (within  former  Rifle  Range  #12). 

4.2.1  SITE  SELECTION 

This  site  was  chosen  as  the  next  in  a  progression  of  increasingly  more  complex  sites  for 
demonstration  of  the  classification  process.  The  first  site  in  the  series,  former  Camp  Sibert,  had 
only  one  target-of-interest  and  item  “size”  was  an  effective  discriminant.  At  this  site,  there  are  at 
least  four  targets-of-interest:  60-mm,  81-mm,  and  4.2-in  mortars  and  2.36-in  rockets.  This 
introduces  another  layer  of  complexity  into  the  process. 

4.2.2  SITE  HISTORY 

Camp  SLO  was  established  in  1928  by  the  State  of  California  as  a  National  Guard  Camp. 
Identified  at  that  time  as  Camp  Merriam,  it  originally  consisted  of  5,800  acres.  Additional  lands 
were  added  in  the  early  1940s  until  the  acreage  totaled  14,959.  From  1943  to  1946,  Camp  SLO 
was  used  by  the  U.S.  Army  for  infantry  division  training  including  artillery,  small  anns,  mortar, 
rocket,  and  grenade  ranges.  According  to  the  Preliminary  Historical  Records  Review  (HRR), 
there  were  a  total  of  27  ranges  and  thirteen  training  areas  located  on  Camp  SLO  during  World 
War  II.  Construction  at  the  camp  included  typical  dwellings,  garages,  latrines,  target  houses, 
repair  shops,  and  miscellaneous  range  structures.  Following  the  end  of  World  War  II,  a  small 
portion  of  the  former  camp  land  was  returned  to  its  former  private  owners.  The  U.S.  Army  was 
making  arrangements  to  relinquish  the  rest  of  Camp  SLO  to  the  State  of  California  and  other 
government  agencies  when  the  conflict  in  Korea  started  in  1950.  The  camp  was  reactivated  at 
that  time. 

The  U.S.  Army  used  the  former  camp  during  the  Korean  War  from  195 1  through  1953  where  the 
Southwest  Signal  Center  was  established  for  the  purpose  of  signal  corps  training.  The  HRR 
identified  eighteen  ranges  and  sixteen  training  areas  present  at  Camp  SLO  during  the  Korean 
War.  A  limited  number  of  these  ranges  and  training  areas  were  used  previously  during  World 
War  II.  Following  the  Korean  War,  the  camp  was  maintained  in  inactive  status  until  it  was 
relinquished  by  the  Army  in  the  1960s  and  1970s.  Approximately  4,685  acres  was  relinquished 
to  the  General  Services  Administration  (GSA)  in  1965.  GSA  then  transferred  the  property  to 
other  agencies  and  individuals  beginning  in  the  late- 1960s  through  the  1980s;  most  of  which  was 
transferred  for  educational  purposes  (California  Polytechnic  State  University  and  Cuesta 
College).  A  large  portion  of  Camp  SLO  (the  original  5,880  acres)  has  been  retained  by  the 
California  National  Guard  (CNG)  and  is  not  part  of  the  FUDS  program. 
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4.2.3  SITE  TOPOGRAPHY  AND  GEOLOGY 


The  former  Camp  SLO  site  consists  mainly  of  mountains  and  canyons  classified  as  grassland, 
wooded  grassland,  woodland,  or  brush.  A  major  portion  of  the  site  is  identified  as  grassland  and 
is  used  primarily  for  grazing.  Los  Padres  National  Forest  (woodland)  is  located  to  the  north- 
northeastern  portion  of  the  site.  During  the  hot  and  dry  summer  and  fall  months,  the  intermittent 
areas  of  brush  occurring  throughout  the  site  become  a  critical  fire  hazard. 

The  underlying  bedrock  within  the  former  Camp  SLO  site  area  is  intensely  folded,  fractured,  and 
faulted.  The  site  is  underlain  by  a  mixture  of  metamorphic,  igneous,  and  sedimentary  rocks  less 
than  200  million  years  old.  Scattered  throughout  the  site  are  areas  of  fluvial  sediments  overlaying 
metamorphosed  material  known  as  Franciscan  melange.  These  areas  are  intruded  by  plugs  of 
volcanic  material  that  comprise  a  chain  of  former  volcanoes  extending  from  the  southwest 
portion  of  the  site  to  the  coast.  Due  to  its  proximity  to  the  tectonic  interaction  of  the  North 
American  and  Pacific  crustal  plates,  the  area  is  seismically  active. 

A  large  portion  of  the  site  consists  of  hills  and  mountains  with  three  categories  of  soils  occurring 
within:  alluvial  plains  and  fans;  terrace  soils;  and  hill/mountain  soils.  Occurring  mainly  adjacent 
to  stream  channels  are  the  soils  associated  with  the  alluvial  plains  and  fans.  The  slope  is  nearly 
level  to  moderately  sloping  and  the  elevation  ranges  from  600  to  1,500  feet.  The  soils  are  very 
deep  and  poorly  drained  to  somewhat  excessively  drained.  Surface  layers  range  from  silty  clay 
to  loamy  sand.  The  terrace  soils  are  nearly  level  to  very  steep  and  the  elevations  ranges  from  600 
to  1,600  feet.  Soils  in  this  unit  are  considered  shallow  to  very  deep,  well  drained,  and 
moderately  well  drained.  The  surface  layer  is  coarse  sandy  loam  to  shaley  loam.  The 
hill/mountain  soils  are  strongly  sloping  to  very  steep.  The  elevation  ranges  from  600  to  3,400 
feet.  The  soils  are  shallow  to  deep  and  excessively  drained  to  well  drained  with  a  surface  layer 
of  loamy  sand  to  silty  clay. 

4.2.4  MUNITIONS  CONTAMINATION 


A  large  variety  of  munitions  have  been  reported  as  used  at  the  former  Camp  SLO.  Munitions 
debris  from  the  following  sources  was  observed  throughout  MRS  05  during  the  2007  SI: 


•  4.2-inch  white  phosphorus  mortar 

•  4.2-inch  mortar  base  plate 

•  3. 5 -inch  rocket 

•  37mm  projectile 

•  75mm  projectile 

•  flares  found  of  newer  metal;  suspected 
from  CNG  activities 


•  105mm  projectile 

•  60mm  mortar 

•  81mm  mortar 

•  Practice  bomb 

•  30  cal.  casings  and  fuzes 


At  the  particular  site  of  this  demonstration,  60-mm,  81 -mm,  and  4.2-in  mortars  and  mortar 
fragments  have  been  observed.  During  the  initial  EM61  Mkll  cart  survey,  two  2.36-in  rockets 
were  found  on  the  surface.  The  excavation  of  two  50’  x  50’  grids  in  October  2008,  as  part  of  the 
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preparatory  activities,  has  confirmed  these  observations  and  provided  infonnation  on  the  depths 
of  munitions  at  this  target  site. 

4.2.5  SITE  GEODETIC  CONTROL  INFORMATION 


The  1 1.8-acre  demonstration  site  is  shown  in  Figure  4-2  as  a  series  of  included  30m  x  30m  cells 
with  a  topographical  map  as  the  background.  The  cells  are  color-coded  based  on  the  data 
collection  systems  that  collected  data  on  them,  tan  color  for  all  systems  and  blue  for  vehicular 
systems  only.  An  interior  area  is  excluded  due  to  rock  outcroppings  and  the  local  slope.  There 
are  three  control  point  monuments  available  near  the  site  for  use  as  GPS  base  station  points 
established  by  Cannon  Associates,  of  San  Luis  Obispo,  CA  referenced  against  nearby 
CAL  TRANS  monument  “A1315”  [8].  The  positions  are  listed  in  Table  4-2.  Figure  4-3  shows 
the  locations  of  monuments  “A1315”  and  “ESTCP”  with  respect  to  the  demonstration  site. 
Monuments  “MM”  and  “BUD”  are  located  within  10m  of  “ESTCP”  and  are  not  shown  for 
clarity.  The  horizontal  datum  for  all  values  is  NAD83.  The  vertical  control  is  referenced  to  the 
NAVD88  datum  and  the  Geoid03  geoid.  Latitude  and  Longitude  are  given  in  degrees  /  minutes  / 
seconds.  Northing  and  Easting  values  are  given  in  UTM  Zone  10  (meters). 


Table  4-2  -  Geodetic  Control  at  the  former  Camp  San  Luis  Obispo  Demonstration  Site 


ID 

Latitude 
(NAD83,  deg) 

Longitude  (deg) 
(NAD83,  deg) 

Elevation 

(m) 

Northing  (UTM 
Zone  10,  m) 

Easting 
(UTM  Zone 
10,  m) 

HAE 

(m) 

ESTCP 

35°  20' 37.77465"  N 

120°  44' 25.95073"  W 

113.69 

3,913,515.94 

705,330.89 

76.01 

MM 

35°  20'  37.53766"  N 

120°  44'  25.89483"  W 

113.54 

3,913,508.67 

705,332.47 

75.81 

BUD 

35°  20' 37.61603"  N 

120°  44'  26.18134"  W 

113.66 

3,913,510.92 

705,325.18 

75.93 

28 


UTM  Northing  (m) 


705,200  705,400  705,600 


Figure  4-2  -  ESTCP  UXO  Classification  Study  demonstration  site  at  the  former 
Camp  San  Luis  Obispo.  The  site  is  shown  as  a  series  of  included  30m  x  30m 
cells.  See  the  text  for  further  discussion. 
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704000 


Figure  4-3  -  Locations  of  two  GPS  control  points  with  respect  to  former  Camp  SLO  demonstration  site 

4.2.6  SITE  CONFIGURATION 

The  demonstration  site  was  configured  as  a  single  11.8  acre  area  as  shown  in  Figure  4-2.  The 
site  spans  a  significant  fraction  of  the  hillside  that  is  the  historical  mortar  target.  The  test  pit  was 
located  near  the  logistics  base  and  the  calibration  strip  was  located  outside  the  inner  fence  line, 
convenient  to  the  site  access  road. 

5.0  TEST  DESIGN 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

Each  demonstration  was  designed  to  be  executed  in  two  stages.  The  first  stage  consisted  of  a 
standard  MTADS  dynamic  survey  of  site.  For  the  APG  demonstration,  the  MTADS 
magnetometer  array  was  the  survey  instrument.  The  details  of  the  magnetometer  survey  can  be 
found  in  Reference  9.  For  the  former  Camp  SLO  demonstration,  the  MTADS  EM61  Midi  array 
was  the  survey  instrument.  The  details  of  the  Mkll  survey  can  be  found  in  Reference  10. 

Anomaly  locations  were  identified  from  the  survey  data  in  a  combination  automated  /  manual 
method  as  described  in  Section  6.2.  A  data  segment  around  each  anomaly  center  was  extracted 
and  analyzed  using  the  UX-Analyze  subsystem  of  the  Oasis  montaj  software  package  as 
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described  in  Section  6.2  to  fit  the  data  to  a  dipole  model  and  extract  the  associated  lit  parameters 
(position,  depth,  equivalent  size).  These  lit  results  constituted  the  source  anomaly  list  for  the 
second  stage  of  each  demonstration. 

This  method  relies  on  the  establishment  of  an  anomaly  detection  threshold.  At  the  former  Camp 
Sibert  demonstration  site,  a  single  munitions  type  was  present.  Pit  measurements  at  various 
depths  and  orientations  of  an  example  article  were  made  and  bounding  response  curves 
generated  for  the  4.2-in  mortar,  the  munitions  of  interest.  The  anomaly  detection  threshold  was 
then  set  based  on  the  least-favorably  predicted  response  at  the  USACoE  standard  llx  depth. 
These  demonstration  sites  each  contained  different  mixes  of  emplaced  munitions  and  suspected 
existing  munitions  contamination.  Individual  anomaly  detection  thresholds  were  established  for 
each  site/area  based  on  sets  of  pit  measurements  made  at  Blossom  Point  for  each  of  the  emplaced 
items.  For  each  site/area,  the  smallest  appropriate  least-favorable  response  will  be  used  to 
detennine  the  threshold.  The  details  of  the  anomaly  selection  process,  including  the  response 
curves  can  be  found  in  the  demonstration  data  reports  [9,10]. 

The  second  stage  of  each  demonstration  was  the  survey  of  each  site/area  using  the  TEMTADS 
array  developed  as  part  of  ESTCP  Project  MM-0601.  The  array  was  positioned  roughly  over  the 
center  of  each  anomaly  on  the  source  anomaly  list  and  a  data  set  collected.  Each  data  set  was 
then  inverted  using  the  data  analysis  methodology  discussed  in  Section  5.6.2,  estimated  target 
parameters  determined,  and  ultimately  a  classification  made  for  each  anomaly.  The  resulting 
prioritized  dig  lists  were  then  submitted  to  either  the  ATC  or  IDA  for  scoring  and  performance 
assessment. 

5.2  SITE  PREPARATION 

Each  demonstration  site  had  been  previously  configured  with  clearly-marked  calibration  and 
open  field  scenarios.  At  least  one  GPS  control  point  was  provided  at  each  site.  Basic  facilities 
such  as  portable  toilets  and  field  buildings  were  provided  at  APG  and  acquired  for  SLO.  Secure 
storage  for  larger  vehicles  and  sensor  arrays  was  limited  at  both  sites.  A  40-foot  shipping 
container  was  mobilized  to  the  site  for  the  duration  of  each  demonstration  to  provide  convenient, 
secure  storage  for  the  MTADS  tow  vehicle  and  the  sensor  trailer.  The  container  was  removed  at 
the  end  of  each  demonstration. 

5.3  SYSTEMS  SPECIFICATION 

This  demonstration  was  conducted  using  the  NRL  MTADS  tow  vehicle  and  subsystems.  The 
tow  vehicle  and  each  subsystem  are  described  further  in  the  following  sections. 

5.3.1  MTADS  Tow  Vehicle 

The  MTADS  has  been  developed  with  support  from  ESTCP.  The  MTADS  hardware  consists  of 
a  low-magnetic-signature  vehicle  that  is  used  to  tow  the  different  sensor  arrays  over  large  areas 
(10  -  25  acres  /  day)  to  detect  buried  UXO.  The  MTADS  tow  vehicle  and  magnetometer  array 
are  shown  in  Figure  5-1. 
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Figure  5-1  -  MTADS  tow  vehicle  and  magnetometer  array. 

5.3.2  RTK  GPS  System 

Positioning  is  provided  using  cm-level  Real  Time  Kinematic  (RTK)  Global  Positioning  System 
(GPS)  receivers.  To  achieve  cm-level  precision,  a  fixed  reference  base  station  is  placed  on  an 
established  first-order  survey  control  point  near  the  survey  area.  The  base  station  transmits 
corrections  to  the  GPS  rover  at  1  Hz  via  a  radio  link  (450  MHz). 

The  TEMTADS  array  is  located  in  three-dimensional  space  using  a  three-receiver  RTK  GPS 
system  shown  schematically  in  Figure  2-3  [11].  The  three-receiver  configuration  extends  the 
concept  of  RTK  operations  from  that  of  a  fixed  base  station  and  a  moving  rover  to  moving  base 
stations  and  moving  rovers.  The  lead  GPS  antenna  (and  receiver,  Main)  receives  corrections 
from  the  fixed  base  station  at  1  Hz.  The  corrected  position  of  the  Main  GPS  antenna  is  reported 
at  10-20  Hz  using  a  vendor-specific  NMEA-0183  message  fonnat  (PTNL,GGK  or  GGK).  The 
Main  receiver  also  operates  as  a  ‘moving  base,’  transmitting  corrections  (by  serial  cable)  to  the 
next  GPS  receiver  (AVR1)  which  uses  the  corrections  to  operate  in  RTK  mode. 

A  vector  (AVR1,  heading  (yaw),  angle  (pitch),  and  range)  between  the  two  antennae  is  reported 
at  10  Hz  using  a  vendor-specific  NMEA-0183  message  format  (PTNL,AVR  or  AVR).  AVR1 
also  provides  ‘moving  base’  corrections  to  the  third  GPS  antenna  (AVR2)  and  a  second  vector 
(AVR2)  is  reported  at  10  Hz.  Ah  GPS  measurements  are  recorded  at  full  RTK  precision,  -2-5 
cm.  The  GPS  position  is  averaged  for  2  seconds  as  part  of  the  data  acquisition  cycle.  The 
averaged  position  and  orientation  information  are  then  recorded  to  the  position  data  file. 

5.3.3  Time-Domain  Electromagnetic  Sensor 

The  TEMTADS  array  is  a  5  x  5  square  array  of  individual  sensors.  Each  sensor  has  dimensions 
of  40  cm  x  40  cm,  for  an  array  of  2  m  x  2  m  overall  dimensions.  The  rationale  of  this  array 
design  is  discussed  in  Reference  12.  The  result  is  a  cross-track  and  down- track  separation  of  40 
cm.  Sensor  numbering  is  indicated  in  Figure  2-3.  The  transmitter  electronics  and  the  data 
acquisition  computer  are  mounted  in  the  tow  vehicle.  Custom  software  written  by  NRL  provides 
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both  navigation  to  the  individual  anomalies  and  data  acquisition  functionality.  After  the  array  is 
positioned  roughly  centered  over  the  center  of  the  anomaly,  the  data  acquisition  cycle  is  initiated. 
Each  transmitter  is  fired  in  a  sequence  winding  outward  from  the  center  position  (12)  in  a 
clockwise  direction.  The  received  signal  is  recorded  for  all  25  Rx  coils  for  each  transmit  cycle. 
The  transmit  pulse  waveform  duration  is  2.7s  (0.9s  block  time,  9  repeats  within  a  block,  3  blocks 
stacked,  with  a  50%  duty  cycle).  While  it  is  possible  to  record  the  entire  decay  transient  at  500 
MHz,  we  have  found  that  binning  the  data  into  115  time  gates  simplifies  the  analysis  and 
provides  additional  signal  averaging  without  significant  loss  of  temporal  resolution  in  the 
transient  decays  [13].  The  data  are  recorded  in  a  binary  fonnat  as  a  single  file  with  25  data 
points  (one  data  point  per  Tx  cycle).  The  filename  corresponds  to  the  anomaly  under 
investigation. 

5.4  CALIBRATION  ACTIVITIES 

For  the  TEMTADS  array,  a  significant  amount  of  data  has  been  collected  with  the  system  as 
configured  at  our  Blossom  Point  facility,  both  on  a  test  stand  and  in  the  towed  configuration  on 
our  test  field  [14].  These  data  and  the  corresponding  fit  parameters  provide  us  with  a  set  of 
reference  parameters  including  those  of  clear  background  (i.e.  no  anomaly  present). 

At  APG,  daily  calibration  efforts  consisted  of  collecting  background  (no  anomaly)  data  sets  at 
the  beginning  and  end  of  each  survey  day  and  periodically  throughout  the  day  at  quiet  spots  to 
determine/monitor  the  system  noise  floor.  A  canonical  reference  object,  a  4”  A1  sphere,  was 
placed  in  the  test  pit  located  near  the  Calibration  Area  and  measured  each  day  to  monitor  the 
daily  variation  in  the  system  response.  These  two  types  of  measurements  constituted  the  daily 
calibration  activities. 

The  sphere  measurements  are  background  subtracted  and  inverted  to  obtain  the  target 
polarizabilities.  These  data  allow  us  to  calibrate  our  sensor  response  in  two  separate  ways.  First, 
by  comparing  the  polarizabilities  derived  from  each  separate  measurement,  we  can  monitor  the 
daily  variation  in  the  system  response.  In  Figure  5-2,  we  plot  the  average  polarizability  for  all 
sphere  measurements  obtained  during  our  entire  run  as  a  function  of  decay  time.  Variations  of 
less  than  10%  of  the  reference  values  were  observed  over  the  course  of  eight  days. 
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Figure  5-2  -  Average  polarizability  for  all  measurements  at  APG  of  the  4”  Aluminum  sphere  as  a 
function  of  decay  time 

Second,  we  compared  our  derived  polarizabilities  directly  with  the  predicted  decay  of  the  sphere. 
The  theoretical  decay  depends  on  the  target’s  size  and  conductivity.  We  have  determined  this 
latter  value  for  the  sphere  by  careful  measurements  with  a  host  of  sensors  over  the  years.  The 
theoretical  decay  is  then  calculated  and  the  derived  polarizabilities  are  scaled  in  amplitude  to 
produce  the  best  match.  Figure  5-3  shows  the  result  for  one  measurement. 

As  this  was  the  first  demonstration  of  the  TEMTADS  array,  the  effect  of  background  variation 
from  point  to  point  and  over  time  was  not  fully  understood  and  remains  an  object  of  study.  As 
such,  frequent  backgrounds  were  taken  over  the  course  of  each  field  day  for  immediate  use  for 
background  subtraction  and  future  analysis  regarding  issues  of  background  variability  and  the 
impact  on  fit  parameter  extraction.  Nine  background  locations  were  identified  from  the 
magnetometer  data  as  being  ‘quiet’  or  having  a  very  low  background  response  level.  The 
selected  locations  were  positioned  throughout  the  site  to  minimize  the  drive  time  to  a  background 
point  from  any  position  on  the  field.  Additional  details  on  system  calibration  at  APG  can  be 
found  in  Reference  9. 
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Figure  5-3  -  Comparison  of  measured  (blue)  versus  theoretical  (red)  polarizability  as  a  function 
of  time  for  the  4”  Aluminum  sphere 

At  the  former  Camp  SLO,  a  calibration  strip  comprised  of  two  replicates  of  each  item  of  interest 
was  emplaced  on  site  to  verily  proper  system  operation  on  a  daily  basis.  The  calibration  strip  was 
surveyed  each  morning  and  each  evening  that  data  are  collected.  The  data  were  preprocessed, 
checked  for  data  quality,  and  then  the  signal  strengths  and  noise  levels  compared  to  the  site- 
specific  response  curves  and  background  levels  to  verify  consistency  of  system  performance. 
Please  refer  to  the  ESTCP  UXO  Classification  Study  Demonstration  Plan  and  the  MTADS 
Magnetometer  /  EM61  MKII  Demonstration  Data  Report  [10]  for  further  details. 

5.5  DATA  COLLECTION 

5.5.1  Scale  of  Demonstration 

The  APG  demonstration  was  conducted  at  the  APG  Standardized  UXO  Test  Site.  A 
magnetometer  survey  was  conducted  on  the  Calibration  and  Blind  Grids,  as  well  as  the  Indirect 
Fire  Area  (approximately  4.3  acres)  on  May  7,  2008.  The  TEMTADS  array  surveyed  the 
Calibration  and  Blind  Grids.  The  array  was  also  deployed  to  approximately  700  anomalies  in  the 
Indirect  Fire  Area  that  were  detected  from  the  magnetometer  data  set.  The  TEMTADS  array 
portion  of  the  demonstration  occurred  from  June  16-23,  2008. 

A  cued  discrimination  survey  of  the  former  Camp  SLO  demonstration  was  conducted  within  the 
11.8  acre  final  demonstration  site  of  approximately  1,500  previously-identified  anomalies  from 
the  anomaly  list  generated  from  the  MTADS  EM61  Mkll  data  set.  This  survey  was  conducted 
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using  the  NRL  TEMTADS.  Characterization  of  the  system  responses  to  the  items  of  interest  was 
determined  using  on  site  measurements  (both  pit  data  and  calibration  strip).  The  data  segment 
(chip)  for  each  anomaly  was  analyzed,  and  dipole  model  fit  parameters  extracted. 

5.5.2  Sample  Density 

Magnetometer  data  were  collected  with  nominal  down-track  spacing  of  6  cm  and  cross  track 
spacing  of  25  cm.  EM61  Mkll  EMI  data  were  collected  with  a  nominal  down-track  spacing  of 
15  cm  and  a  cross  track  spacing  of  50  cm.  Two  orthogonal  surveys  were  conducted  to  increase 
target  illumination  and  data  density.  The  EMI  data  spacing  for  the  TEMATDS  is  fixed  at  40  cm 
in  both  directions  by  the  array  design. 

5.5.3  Quality  Checks 

Preventative  maintenance  inspections  were  conducted  continually  by  all  team  members  during 
data  collection,  focusing  particularly  on  the  tow  vehicle  and  sensor  trailer.  Deficiencies  were 
addressed  according  to  the  severity  of  the  deficiency.  Parts,  tools,  and  materials  for  many 
maintenance  scenarios  are  available  in  the  system  spares  inventory,  a  fraction  of  which  was  on 
site  at  APG  due  to  the  proximity  of  our  base  of  operations  at  Blossom  Point,  MD.  Please  refer  to 
the  Demonstration  Data  Reports  for  further  details  on  the  survey  instrument  QC  procedures. 

Since  the  TEMTADS  operates  in  a  cued  mode,  the  data  QC  procedures  and  checks  differ  from 
that  of  the  survey  mode  instruments.  The  status  of  the  RTK  GPS  system  can  be  visually 
determined  by  the  operator  prior  to  starting  the  data  collection  cycle,  assuring  that  the  position 
and  orientation  information  are  valid  (FQ  3)  during  the  collection  period. 

Two  data  quality  checks  were  perfonned  on  the  TEMTADS  data.  After  background  subtraction, 
contour  plots  of  the  signal  were  generated  for  the  25  transmit/receive  pairs  at  a  decay  time  of 
0.042  ms.  An  example  of  a  good  data  set  from  a  single  anomaly  with  a  large  SNR  is  shown  in 
Figure  5-4  for  Calibration  Area  item  16.  The  plots  were  visually  inspected  to  verify  that  there 
was  a  well  defined  anomaly  without  extraneous  signals  or  dropouts.  QC  on  the  transmit/receive 
cross  terms  was  based  on  the  dipole  inversion  results.  Our  experience  has  shown  that  data 
glitches  show  up  as  reduced  dipole  fit  coherence.  No  data  glitches  were  observed  during  our 
APG  demonstration,  so  data  collected  at  our  Blossom  Point  facility  for  a  105mm  HEAT 
Projectile  are  used  as  an  example.  The  fit  polarizabilities  as  a  function  of  decay  time  are  shown 
with  all  elements  (Figure  5-5)  and  with  element  21  excluded  (Figure  5-6),  with  fit  coherences  of 
0.699  and  0.992  respectively.  In  comparing  Figure  5-5  and  Figure  5-6,  the  expected  axial 
symmetry  of  the  minor  polarizabilities  is  restored  with  the  exclusion  of  element  2 1  in  this  case. 
The  problem  with  element  2 1  is  only  observable  at  late  time  and  therefore  would  not  be  captured 
in  the  monostatic  plot  review. 
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Figure  5-4  -  Monostatic  QC  contour  plot  for  Calibration  Area  item  16 
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Figure  5-5  -  Polarizability  as  a  function  of  time  for  a  1 05mm  HEAT  Projectile  with  all  data 
included.  The  fit  coherence  for  all  elements  included  was  0.699. 
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Figure  5-6  -  Polarizability  as  a  function  of  time  for  a  1 05mm  FIEAT  Projectile  with  element  21 
excluded.  The  fit  coherence  with  element  21  excluded  was  0.992. 

The  issue  of  when  multiple  objects  are  found  to  be  under  the  array  simultaneously,  generating 
overlapping  signatures,  can  also  be  addressed  at  this  point  in  the  data  QC  process.  An  example 
case  is  shown  in  Figure  5-7.  There  are  two  apparent  issues  in  the  data  set.  First,  there  appears  to 
be  a  small,  shallow  bit  of  scrap  on  top  of  the  target.  Second,  there  was  a  bit  of  scrap  present  in 
the  background  file  used.  This  latter  issue  is  seen  in  the  data  from  element  0.  Figure  5-8  shows 
the  fit  betas  obtained  using  all  of  the  data.  Figure  5-9  shows  the  fit  betas  obtained  excluding 
elements  0,5,10,11,15,16,17,20,21,22.  The  latter  betas  match  our  library  105mm  HEAT  betas 
very  well. 

Any  data  set  that  was  deemed  unsatisfactory  by  the  data  analyst  was  flagged  and  not  processed 
further.  The  anomaly  corresponding  to  the  flagged  data  was  logged  for  future  re-acquisition. 
Data  which  meet  these  standards  are  of  the  quality  typical  of  the  MTADS  system. 
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Figure  5-7  -  Monostatic  QC  contour  plot  for  example  anomaly 
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Figure  5-8  -  Polarizability  as  a  function  of  time  for  the  example  show  in  Figure  5-7  with  all  data 
included.  The  fit  coherence  for  all  elements  included  was  0.652. 


39 


100.0000 

10.0000 


1.0000 

</3 

|  0.1000 
CD 

0.0100 


0.0010 

0.0001 

0.01  0.10  1.00  10.00  100.00 

Time(ms) 

Figure  5-9  -  Polarizability  as  a  function  of  time  for  the  example  show  in  Figure  5-7  with  elements 
0,5,10,11,15,16,17,20,21,22  excluded.  The  fit  coherence  for  the  remaining  elements  was  0.985. 

5.5.4  Data  Handling 

Data  were  stored  electronically  as  collected  on  the  MTADS  vehicle  data  acquisition  computer 
hard  drives.  Approximately  every  two  survey  hours,  the  collected  data  were  copied  onto 
removable  media  and  transferred  to  the  data  analyst  for  QC/analysis.  The  data  were  moved  onto 
the  data  analyst’s  computer  and  the  media  recycled.  Raw  data  and  analysis  results  were  backed 
up  from  the  data  analyst’s  computer  to  optical  media  (CD-R  or  DVD-R)  or  external  hard  disks 
daily.  These  results  are  archived  on  internal  file  servers  at  NRL  or  SAIC.  All  field  notes  / 
activity  logs  were  written  in  ink  and  stored  in  archival  laboratory  notebooks.  These  notebooks 
are  archived  at  NRL  or  SAIC.  Relevant  sections  are  reproduced  in  demonstration  reports  such  as 
this.  Dr.  Tom  Bell  is  the  POC  for  obtaining  data  and  other  information.  His  contact  information 
is  provided  in  Appendix  A  of  this  report. 

5.5.5  APG  Discrimination  Survey  Data  Summary 

The  MTADS  tow  vehicle  and  TEMTADS  array  were  mobilized  to  the  APG  Test  Site  during  the 
week  of  June  9,  2008.  The  TEMTADS  array  portion  of  the  demonstration  commenced  on  June 
16,  2008  with  a  data  collection  in  the  Calibration  Area.  Cued  surveys  were  conducted  of  the 
Calibration,  Blind  Grid,  and  the  Indirect  Fire  Area.  Data  collection  was  completed  on  June  23, 
2008.  The  system  was  demobilized  later  that  week  to  our  Blossom  Point  facilities. 
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The  array  was  positioned  roughly  over  the  center  of  each  anomaly  on  the  source  anomaly  list  for 
each  area  and  a  data  set  collected.  Each  data  set  was  then  inverted  using  the  data  analysis 
methodology  discussed  in  Section  5.6.2,  estimated  target  parameters  detennined,  and  ultimately 
a  classification  made  for  each  item  for  the  Blind  Grid  and  Indirect  Fire  Area. 

5.5.5.1.  Calibration  Area 

The  ground  truth  for  the  Calibration  Area  is  known  and  the  collected  data  were  used  to  verify  the 
operational  status  of  our  systems  and  processing  procedures.  Each  of  the  66  seed  locations  in  the 
Calibration  Area  was  investigated  in  turn  and  the  resultant  fit  parameters  used  to  cross-check 
against  and  augment  our  library  of  responses  to  known  munitions  items.  Data  collection 
occurred  on  June  16,  2008. 

5.5.5.2. Blind  Grid 

The  Blind  Grid  is  composed  of  400  cells  in  a  20x20  array  with  2m  on-center  spacing.  The  Rank 
1  and  2  anomalies  (214  anomalies)  identified  in  the  magnetometer  survey  (See  Reference  9) 
formed  the  Blind  Grid  anomaly  list  for  the  TEMTADS  array.  ATC  staff  placed  a  plastic  pin  flag 
on  the  center  of  each  of  the  214  anomalies  investigated  prior  to  this  portion  of  the  demonstration. 
Data  collection  occurred  during  the  period  June  16  -  17,  2008.  The  background-corrected  data 
set  for  each  anomaly  is  included  on  the  accompanying  DVD. 

5.5.5.3.Indirect  Fire  Area 

The  Indirect  Fire  Area  is  a  1.3  ha  area  drawn  from  the  original  configuration’s  Open  Field.  The 
IDF  has  been  seeded  with  three  munitions  types,  the  60mm  and  81mm  Mortar  and  the  105mm 
Projectile.  The  694  anomalies  in  Ranks  1,  2,  3,  and  7  (See  Reference  9)  were  added  to  the  IDF 
anomaly  list  for  the  TEMTADS  array.  Data  collection  occurred  during  the  period  June  17  -  23, 
2008. 

5.5.6  Former  Camp  San  Luis  Obispo 

The  MTADS  tow  vehicle  and  the  TEMTADS  array  were  mobilized  to  the  former  Camp  SLO 
along  with  the  other  MTADS  systems  in  May,  2009.  The  TEMTADS  portion  of  the 
demonstration  commenced  on  June  8,  2009.  Cued  surveys  were  conducted  for  approximately 
1,500  anomalies  identified  from  the  MTADS  EM61  Mkll  array  survey  data.  The  Calibration 
Strip  was  surveyed  twice  daily,  once  at  the  beginning  and  once  at  the  end  of  the  survey  day. 
Data  collection  was  completed  on  June  18,  2009.  The  system  was  demobilized  later  that  week  to 
our  Blossom  Point  facilities. 

The  array  was  positioned  roughly  over  the  center  of  each  anomaly  on  the  source  anomaly  list  and 
a  data  set  collected.  Each  data  set  was  then  inverted  using  the  data  analysis  methodology 
discussed  in  Sections  6.1  and  6.3,  estimated  target  parameters  determined,  and  ultimately  a 
classification  made  for  each  entry  on  the  anomaly  list. 
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5.5.6.I. Calibration  Strip 


A  calibration  strip,  comprised  of  two  samples  of  each  of  the  four  targets  of  interest  plus  two 
shotputs,  was  emplaced  on  site  as  a  means  of  verifying  proper  system  operation  on  a  daily  basis. 
The  strip  was  surveyed  twice  daily,  once  at  the  beginning  of  the  day,  and  once  at  the  end.  The 
sole  exception  to  this  procedure  occurred  on  our  third  day,  when  brake  problems  with  the  vehicle 
forced  an  early  cessation  of  activities.  The  measured  peak  signal  for  each  emplaced  item  was 
determined  for  each  run  and  compared  over  the  course  of  the  demonstration  to  insure  the 
reproducible,  physically  realistic  performance  of  the  sensor  array.  The  results  are  discussed  in 
Section  7.0. 

5.5.6.2.Main  Demonstration 

The  former  Camp  SLO  demonstration  site  was  1 1.8  acre  area  as  shown  in  Figure  4-2.  A  total  of 
1547  anomalies  (including  reacquisitions  requested  by  the  Data  Analyst)  were  measured  over  a 
period  of  10  days  for  an  average  of  155  anomalies/day.  The  only  days  for  which  our  average  fell 
below  our  goal  of  125  anomalies/day  were  the  final  day,  due  to  completing  the  data  collection, 
and  2  other  days  in  which  necessary  vehicle  maintenance  shortened  our  workday. 

5.6  VALIDATION 

5.6.1  Aberdeen  Proving  Grounds 

With  the  exception  of  the  Calibration  Grid,  the  ground  truth  for  the  Standardized  sites  is  held 
back  from  individual  technology  demonstrators  to  preserve  the  utility  of  the  Blind  Grid  and  Open 
Field  Areas.  Results  from  the  Blind  Grid  and  the  Indirect  Fire  Area  were  submitted  to  ATC  for 
performance  evaluation.  Scoring  results  have  been  received  and  are  available  [15].  A  summary 
of  the  results  are  given  in  Section  7.0. 

5.6.2  Former  Camp  San  Luis  Obispo 

At  the  conclusion  of  data  collection  activities,  all  anomalies  on  the  master  anomaly  list 
assembled  by  the  Program  Office  were  excavated.  Each  item  encountered  was  identified, 
photographed,  its  depth  measured,  its  location  determined  using  cm-level  GPS,  and  the  item 
removed  if  possible.  All  non-hazardous  items  were  saved  for  later  in-air  measurements  as 
appropriate.  This  ground  truth  information,  once  released,  was  used  to  validate  the  objectives 
listed  in  Section  3.2. 

6.0  DATA  ANALYSIS  AND  PRODUCTS 

The  TEMTADS  is  a  cued  UXO  classification  system.  There  were  two  parts  to  the  data 
collection  in  each  demonstration:  an  initial  survey  which  served  primarily  to  locate  targets,  and 
TEMTADS  measurements  made  over  the  each  target  detected  in  the  survey.  The  TEMTADS 
data  were  used  for  target  classification  and  discrimination. 
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6.1  PREPROCESSING 


The  collected  survey  data  were  preprocessed  on  site  for  quality  assurance  purposes  using 
standard  TEMTADS  procedures  and  checks  as  discussed  in  Section  5.5.3  and  References  9  and 
19. 

The  TEMTADS  has  25  transmitters/receiver  pairs.  For  each  transmit  pulse,  the  response 
measured  at  all  of  the  receivers  was  recorded  simultaneously.  For  each  target,  a  25  x  25  x  N  data 
array  is  generated,  where  N  is  the  number  of  recorded  time  gates.  The  current  system 
configuration  bins  the  data  in  121  logarithmically  spaced  gates.  During  the  preprocessing  step, 
the  recorded  signals  are  normalized  by  the  transmitter  currents  to  account  for  any  transmitter 
variations.  To  account  for  time  delay  due  to  effects  of  the  receive  coil  and  electronics,  we 
subtract  0.028  ms  from  the  nominal  gate  times  [16].  The  delay  was  previously  determined 
empirically  by  comparing  measured  responses  for  test  spheres  with  theory.  The  measured 
responses  include  distortions  due  to  transmitter  ringing  and  related  artifacts  out  to  about  40  ps. 
Consequently  we  only  included  the  responses  from  decay  times  beyond  40  ps  in  our  analyses. 
This  leaves  115  gates  spaced  logarithmically  between  0.042  ms  and  25  ms. 

The  background  response  was  subtracted  from  each  target  measurement  using  data  collected  in  a 
nearby  target-free  region,  or  background.  Background  locations  were  selected  from  quiet  areas 
observed  in  the  magnetometer  anomaly  map.  All  of  the  background  measurements  were  inter- 
compared  to  evaluate  background  variability  and  to  identify  outlier  measurements  potentially 
corresponded  to  measurements  over  non-ferrous  targets.  We  have  not  observed  significant 
background  variability  in  our  measurements  at  our  Blossom  Point  test  site,  and  were  able  to  use 
blank  ground  measurements  from  100  meters  away  for  background  subtraction  on  targets  in  the 
test  field. 

Geo-referencing  of  the  array  data  is  based  on  the  GPS  data,  which  gives  the  location  of  the  center 
of  the  array  and  the  orientation  of  the  array.  Sensor  locations  within  the  array  are  fixed  by  the 
array  geometry.  Dipole  inversion  of  the  array  data  (Section  6.3)  determines  target  location  in 
local  array-based  coordinates.  The  local  position  was  then  transformed  to  absolute  coordinates 
using  the  array  location  and  orientation  determined  from  the  corresponding  GPS  data. 

6.2  TARGET  SELECTION  FOR  DETECTION 

Targets  were  selected  from  the  survey  data  using  the  threshold  exceedance  methods  described  in 
References  9  and  10  using  a  physics-based  threshold  in  Oasis  montaj.  The  data  chips  associated 
with  the  detected  anomalies  were  then  processed  in  an  automatic  processing  mode  of  UX- 
Analyze.  The  results  of  the  automatic  processing  were  then  reviewed  in  UX-Analyze’s 
interactive  mode  to  allow  operator  experience  to  be  included  in  the  target  selection.  In  the  case 
of  the  APG  demonstration,  a  small  number  of  additional  anomalies  were  identified  by  the 
operator  and  added  to  the  anomaly  list.  For  the  former  Camp  SLO  demonstration,  the  operator 
coalesced  double-peaked  features  into  a  single  anomaly  where  appropriate. 
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The  UX-Analyze  environment  for  anomaly  review  contains  two  main  graphics  tools.  One 
displays  an  interpolated  amplitude  image  of  all  the  data  processed  at  the  site  for  the  current 
survey  and  allows  the  analyst  to  see  which  grids  of  the  site  have  been  surveyed  and  to  select 
grids  for  target  analysis.  The  area  selected  for  analysis  is  displayed  in  another  graphic  window 
in  which  the  analyst  visually  selects  targets  for  review.  In  both  of  these  windows  there  are 
options  for  changing  the  view  area,  the  mapping  color  palette  and  contrast,  the  amount  of 
information  overlaid  on  the  screen,  etc.  To  refine  a  characterization  of  a  target,  the  analyst 
modifies  the  data  boundary  using  the  mouse.  When  the  boundary  is  closed,  the  data  inside  are 
sent  to  a  dipole  fitting  algorithm  to  detennine  target  parameters  such  as  location,  depth  and 
apparent  size.  The  anomaly  lists  were  then  sorted  prioritized  dig  lists. 

6.3  PARAMETER  ESTIMATION 

The  raw  signature  data  from  the  TEMTADS  reflect  details  of  the  sensor/target  geometry  as  well 
as  inherent  EMI  response  characteristics  of  the  targets  themselves.  In  order  to  separate  out  the 
intrinsic  target  response  properties  from  sensor/target  geometry  effects  we  invert  the  signature 
data  to  estimate  principal  axis  magnetic  polarizabilities  for  the  targets.  The  TEM  data  are 
inverted  using  the  standard  induced  dipole  response  model  wherein  the  effect  of  eddy  currents 
set  up  in  the  target  by  the  primary  field  is  represented  by  a  set  of  three  orthogonal  magnetic 
dipoles  at  the  target  location  [17].  The  measured  signal  is  a  linear  function  of  the  induced  dipole 
moment  m,  which  can  be  expressed  in  terms  of  a  time  dependent  polarizability  tensor  B  as 

m  =  UBUTH0 

where  U  is  the  transformation  matrix  between  the  physical  coordinate  directions  and  the 
principal  axes  of  the  target  and  Ho  is  the  primary  field  strength  at  the  target.  The  eigenvalues  Pi(t) 
of  the  polarizability  tensor  are  the  principal  axis  polarizabilities. 

Given  a  set  of  measurements  of  the  target  response  with  varying  geometries  or  "look  angles"  at 
the  target,  the  data  can  be  inverted  to  determine  the  (X,Y,Z)  location  of  the  target,  the  orientation 
of  its  principal  axes  ((j),0,\|/),  and  the  principal  axis  polarizabilities  (Pi,p2,P3).  The  basic  idea  is  to 
search  out  the  set  of  nine  parameters  (X,Y,Z,(j),0,\|/,Pi,P2,P3)  that  minimizes  the  difference 
between  the  measured  responses  and  those  calculated  using  the  dipole  response  model. 

For  the  TEMTADS  data,  inversion  is  accomplished  by  a  two-stage  method.  In  the  first  stage,  the 
target’s  (X,Y,Z)  dipole  location  is  solved  for  non-linearly.  At  each  iteration  within  this 
inversion,  the  nine  element  polarizability  tensor  (B)  is  solved  linearly.  We  require  that  this 
tensor  be  symmetric;  therefore,  only  six  elements  are  unique.  Initial  guesses  for  X  and  Y  are 
determined  by  a  signal-weighted  mean.  The  routine  normally  loops  over  a  number  of  initial 
guesses  in  Z,  keeping  the  result  giving  the  best  fit  as  measured  by  the  chi-squared  value.  The 
non-linear  inversion  is  done  simultaneously  over  all  time  gates,  such  that  the  dipole  (X,Y,Z) 
location  applies  to  all  decay  times.  At  each  time  gate,  the  eigenvalues  and  angles  are  extracted 
from  the  polarizability  tensor. 
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In  the  second  stage,  six  parameters  are  used:  the  three  spatial  parameters  (X,Y,Z)  and  three 
angles  representing  the  yaw,  pitch,  and  roll  of  the  target  (Euler  angles  (j),0,i|/).  Here  the 
eigenvalues  of  the  polarizability  tensor  are  solved  for  linearly  within  the  6-parameter  non-linear 
inversion.  In  this  second  stage  both  the  target  location  and  its  orientation  are  required  to  remain 
constant  over  all  time  gates.  The  value  of  the  best  fit  X,  Y,  and  Z  from  the  first  stage,  and  the 
median  value  of  the  first-stage  angles  are  used  as  an  initial  guess  for  this  stage.  Additional  loops 
over  depth  and  angles  are  included  to  better  ensure  finding  the  global  minimum. 

Figure  6-1  shows  an  example  of  the  principal  axis  polarizabilities  determined  from  TEMTADS 
data.  The  target,  a  mortar  fragment,  is  a  slightly  bent  plate  about  Vi  cm  thick,  25  cm  long  and  15 
cm  wide.  The  red  curve  is  the  polarizability  when  the  primary  field  is  normal  to  the  surface  of 
the  plate,  while  the  green  and  blue  curves  correspond  to  cases  where  the  primary  field  is  aligned 
along  each  of  the  edges. 


Time  (msec) 

Figure  6-1  -  Principal  axis  polarizabilities  for  a  Vi  cm  thick  by  25cm  long 
by  15  cm  wide  mortar  fragment. 

Not  every  target  from  the  detection  surveys  had  a  strong  enough  TEM  response  to  support 
extraction  of  target  polarizabilities.  All  of  the  data  were  run  through  the  inversion  routines,  and 
the  results  were  manually  screened  to  identify  those  targets  that  could  not  be  reliably  classified. 
Several  criteria  were  used  in  this  process:  signal  strength  relative  to  background,  dipole  fit  error 
(difference  between  data  and  model  fit  to  data),  the  visual  appearance  of  the  polarizability 
curves. 

For  moderate-to-good  SNR,  our  derived  polarizabilities  can  be  directly  compared  with  those  of 
our  target  library.  In  these  cases,  a  metric  was  computed  based  on  how  well  the  amplitude  and 
ratio  of  the  target  polarizabilities  match  those  of  library  objects.  The  metric  runs  from  0  (no 
match)  to  1  (perfect  match).  For  the  APG  demonstration  and  targets  with  lower  SNR,  or  for 
cases  where  a  visual  inspection  suggested  a  match,  but  the  match  quality  was  below  our  cutoff, 
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target  classification  and  discrimination  were  done  using  library  matching  procedures  similar  to 
those  used  by  Sky  Research,  Lawrence  Berkeley  Lab  and  ourselves  in  the  ESTCP  Discrimination 
Study  Pilot  Program  at  the  former  Camp  Sibert  [18].  The  fit  quality  of  an  unconstrained  dipole 
inversion  of  the  TEMTADS  data  was  compared  to  the  fit  quality  of  a  dipole  fit  constrained  by  a 
set  of  principal  axis  polarizabilities  drawn  from  a  signature  library.  Fit  quality  is  the  squared 
correlation  coefficient  between  the  model  fit  and  the  data.  If  the  ratio  of  the  constrained  fit 
quality  to  the  unconstrained  fit  quality  (p)  is  one,  then  the  library  item  is  as  good  a  match  to  the 
data  as  possible.  If  the  ratio  is  small,  then  the  library  item  is  a  poor  match. 

The  scoring  rules  for  the  SLO  demonstration  were  different  than  those  used  for  APG 
submissions,  including  the  introduction  of  the  “Can’t  Decide”  category  discussed  in  the  next 
section.  Therefore  a  similar,  but  modified  process  was  used.  The  primary  library  match 
algorithm  compares  the  polarizabilities  of  an  unknown  target  with  each  library  entry  based  on  3 
criteria:  the  amplitude  of  Pi,  and  the  two  shape  parameters,  p2/pi  and  p3/pi-  The  difference  in  the 
values  is  computed  at  all  timegates,  excluding  those  where  the  values  are  negative.  The 
differences  are  then  plugged  into  a  Gaussian.  The  “time  constants”  in  these  Gaussians  were 
derived  by  examining  the  variability  in  the  amplitude  and  shape  parameters  for  a  large  number  of 
objects  for  which  ground  truth  was  known.  Medians  are  taken  to  avoid  bad  data  points.  Finally, 
the  results  from  the  3  different  criteria  are  averaged,  producing  a  metric  which  ranges  from  0 
(worst  possible  fit)  to  1  (perfect  fit).  Note  that  the  procedure  just  described  is  not  a  library 
constrained  match,  i.e.,  we  do  not  invert  our  data  forcing  the  P’s  to  be  those  of  each  library  object 
in  turn,  but  rather  simply  compare  our  unconstrained  polarizabilities  to  those  of  the  library.  As 
such,  the  comparison  runs  rapidly,  and  there  is  no  need  to  reduce  the  number  of  separate  types  in 
the  library  to  balance  computation  time. 

For  each  anomaly,  this  match  is  performed  on  both  the  polarizabilities  from  the  single  dipole  fit, 
and  those  of  each  target  obtained  by  the  multi-dipole  fit.  The  metric  is  set  to  whichever  set  of  P’s 
produces  the  best  match. 

Due  to  the  occasional  “splitting”  of  the  two  smaller  P’s,  we  also  considered  a  second  metric  in 
which  the  3rd  comparison,  namely  the  p3/pi  shape  match,  was  dropped  from  the  calculation.  Two 
separate  prioritized  dig  lists  were  submitted,  one  using  the  3  criteria  match  metric,  and  one  using 
this  2  criteria  match  metric. 

6.4  CLASSIFIER  AND  TRAINING 

Prior  to  the  demonstrations  we  collected  training  data  in  air  for  all  of  the  14  standard  APG 
ordnance  targets  and  the  four  known  munitions  types  for  the  fonner  Camp  SLO  site.  These  data 
were  used  to  generate  the  fit  library  entries.  Many  of  the  targets  are  composites  of  two  or  more 
distinct  parts,  like  a  steel  body  combined  with  an  aluminum  tail  assembly.  Depending  on  the 
distance  between  the  sensors  and  the  target,  such  items  can  exhibit  a  range  of  slightly  different 
EMI  signatures  corresponding  to  excitation  from  different  directions.  We  included 
measurements  with  the  target  oriented  nose  up,  towards  the  sensor  array,  nose  down,  away  from 
the  array,  flat  and  obliquely  in  the  fit  library. 
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Our  experience  at  our  Blossom  Point  test  site  has  been  that  polarizabilities  determined  from  in¬ 
air  measurements  are  indistinguishable  from  those  determined  from  measurements  taken  over 
buried  targets.  We  used  the  data  collected  from  the  site  calibration  areas,  which  contains  several 
instances  of  each  target,  to  establish  that  this  held  true. 

We  have  assembled  a  fairly  extensive  polarizability  database  for  clutter  items  recovered  from 
several  different  sites.  This  library  was  used  as  training  data  for  establishing  UXO/clutter 
discrimination  boundaries  on  the  direct  match  metric  and  on  the  coherence  ratio  p. 

As  noted  in  Section  6.3,  for  anomalies  with  moderate-to-good  SNR  and  no  serious  overlap 
issues,  classification  was  done  using  the  direct  match  metric,  wherein  the  unconstrained 
polarizabilities  are  compared  with  those  of  library  objects.  For  the  APG  demonstration,  using  the 
results  of  the  Calibration  Grid,  and  even  considering  Blind  Grid  and  IDF  anomalies  where  the 
visual  match  to  a  library  object  is  so  good  that  we  have  high  confidence  in  the  identification,  we 
detennined  a  cutoff  value  for  our  metric  of  0.56  (reduced  to  0.44  for  the  25  mm),  above  which 
the  target  is  considered  ordnance,  and  below  which  it  is  considered  clutter.  This  can  be 
considered  a  first  pass  for  classification,  and  was  applied  to  all  anomalies. 

For  the  former  Camp  SLO  demonstration,  a  different  classification  scheme  was  used.  A  series  of 
decision  rules  were  used  to  classifying  anomalies  into  4  categories  for  placement  on  the  final, 
prioritized  dig  list.  These  decision  rules  are  outlined  below,  as  well  as  the  criteria  for  detennining 
overlapping  signatures. 

6.4.1  Cannot  Analyze 

We  classified  targets  as  “Cannot  Analyze”  for  those  which  the  inversion  produced  unphysical 
parameters,  specifically,  depths  below  a  cutoff  based  on  the  largest  target  expected  on  the  site, 
and  negative  polarizabilities.  A  depth  value  of  2m  was  used. 

6.4.2  Likely  Munition 

We  classified  targets  as  “Likely  Munitions”  based  on  the  library  match  metrics  previously 
described.  The  cutoff  is  determined  to  insure  that  all  training  targets  revealed  as  UXO  or  TOI 
with  moderate-to-good  signal-to-noise  were  included.  Our  cutoff  values  were  0.8336  for  the  3 
criteria  metric,  and  0.8925  for  the  2  criteria  metric. 

6.4.3  Cannot  Decide 

Targets  in  this  category  were  comprised  of  four  distinct  types.  The  first  type  consisted  of  targets 
with  very  low  signal-to-noise,  based  on  previous  experience  with  this  instrument.  A  cutoff  signal 
amplitude  value  of  2mv/A  was  used.  The  second  type  was  targets  which  suffered  from  serious 
overlap  issues  with  neighboring  anomalies,  to  the  extent  that  we  were  not  confident  in  our  ability 
to  extract  meaningful  features.  The  remaining  two  categories  excluded  cases  from  the  first  two. 
The  third  category  consisted  of  targets  whose  match  metric  fell  within  a  “buffer  zone”  below  the 
“Likely  Munition”  metric  cutoff.  The  anomalies  in  this  category  possessed  a  match  metric  which 
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was  not  sufficiently  high  to  justify  their  being  classified  as  “Likely  Munitions”,  but  which  is  not 
sufficiently  low  to  classify  them  as  “Likely  Clutter”.  The  actual  values  for  this  lower  metric 
cutoff  were  determined  from  a  visual  inspection  of  matches  in  the  Test  Set  data,  and  were  equal 
to  0.7365  for  the  3  criteria  metric,  and  0.8335  for  the  2  criteria  metric.  Finally,  the  fourth  type 
consisted  of  anomalies  for  which  the  metric  fell  below  the  “Likely  Munition”  cutoff,  but  which 
possess  axial  symmetry.  This  choice  was  made  to  take  account  of  the  fact  that  our  library  is 
finite.  Exceptions  to  the  axial  symmetry  rule  were  targets  that  showed  rapid  decay  (based  on  the 
most  rapid  decay  for  UXO  in  the  complete  library),  and  targets  whose  polarizabilities  were  of 
very  small  amplitude  (based  on  the  smallest  amplitude  UXO  in  the  complete  library).  We 
conservatively  defined  axial  symmetry  as  a  median  agreement  of  50%  between  the  two  smaller 
betas. 

6.4.4  Likely  Clutter 

Targets  were  placed  in  this  category  whose  metric  fell  below  that  of  the  second  “buffer  zone” 
cutoff  and  which  did  not  show  axial  symmetry  as  previously  defined. 

6.4.5  Ranking 

The  “Likely  Clutter”  category  was  divided  into  two  subcategories:  low  signal-to-noise  and 
moderate  to  high  signal-to-noise.  Within  each  of  these  subcategories,  ranking  was  done  by  the 
library  match  metric,  such  that  the  lower  the  metric,  the  higher  the  rank.  However,  all  anomalies 
in  the  low  SNR  subcategory  were  ranked  lower  than  those  in  the  moderate  to  high  SNR 
subcategory.  The  rationale  behind  this  is  that  low  SNR  can  decrease  the  quality  of  the  match 
metric,  and  the  targets  for  which  we  had  the  highest  confidence  of  being  clutter  are  those  with  a 
low  metric  and  decent  SNR.  Similarly,  the  “Cannot  Decide”  category  was  divided  into  three 
subcategories  based  on  the  four  types  of  entries,  with  the  targets  having  low  signal  and  overlap 
issues  having  the  highest  rank,  followed  by  those  with  moderate-to-high  signal  and  no  serious 
overlap  issues,  with  those  targets  having  a  metric  above  the  “buffer  zone”  cutoff  giving  a  lower 
ranking  (less  likely  to  be  clutter)  than  those  below.  The  ranking  within  each  subcategory  was 
ordered  by  the  metric.  Finally,  the  “Likely  Munition”  category  was  ranked  solely  by  the  metric, 
with  the  largest  rank  going  to  the  largest  metric  value. 

Since  it  is  not  obvious  how  to  incorporate  either  the  axial  symmetry  parameter,  the  low  SNR 
parameter,  or  the  overlap  into  our  probabilities,  we  simply  set  the  probability  equal  to  the  1 
minus  the  metric  value  for  all  targets.  Unfortunately,  with  the  exception  of  the  “Likely  Munition” 
category,  this  means  that  there  was  in  general  not  a  1  to  1  mapping  between  the  ranks  and  the 
probabilities. 

6.4.6  Determination  of  Overlapping  Signals 

As  discussed  in  Section  5.5.3,  monostatic  contour  plots  were  made  of  each  anomaly  both  as  a 
QC  check  and  as  a  means  of  determining  whether  the  anomaly  suffers  from  overlapping 
signatures.  Thus,  an  entry  was  made  in  a  spreadsheet  giving  some  indication  of  the  extent  of  the 
overlap,  if  any.  In  some  cases,  this  overlap  may  have  been  from  multiple  targets  within  the 


48 


anomaly,  as  opposed  to  being  caused  by  a  separate  anomaly.  Therefore,  we  used  this  entry  in 
conjunction  with  the  knowledge  of  the  location  of  other  anomalies  relative  to  the  target  of 
interest.  The  edge  of  the  TEMTADS  is  lm  from  a  target  located  under  the  array  center.  Thus,  we 
declaring  any  anomaly  for  which  the  spreadsheet  entry  indicates  overlap,  and  for  which  at  least 
one  target  was  within  1.5m  of  the  anomaly,  as  an  overlapping  signal. 

6.5  DATAPRODUCTS 

The  data  product  requirements  for  the  two  demonstrations  were  different  and  different  data 
products  were  correspondingly  generated.  For  the  APG  demonstration,  we  used  the  standard 
reporting  templates  for  the  Blind  Grid  and  the  Open  Field.  The  metrics  in  Section  3.0  were 
calculated  directly  from  the  Scoring  Report  provided  by  the  Standardized  Test  Site 
administrators.  The  Response  Stage  value  was  the  maximum  signal  level  above  background 
from  the  magnetometer  anomaly.  The  calculation  of  the  discrimination  stage  was  more 
involved.  For  those  targets  classified  as  ordnance,  the  direct  match  metric  was  used  as  the 
ranking.  Additionally,  targets  which  we  could  not  analyze  as  described  in  Section  6.3  were 
classified  as  ordnance,  but  given  the  lowest  confidence  rating,  and  noted  as  such  in  our 
submission.  For  targets  classified  as  clutter,  a  division  was  made  between  targets  that  had 
moderate -to-good  SNR  and  little  to  no  overlap  with  adjacent  targets,  and  those  with  either  low 
SNR  or  more  serious  overlap  issues.  For  the  former  class,  ranking  was  based  on  the  direct  match 
metric,  while  for  the  latter,  ranking  was  based  on  a  combination  of  the  direct  match  metric  and 
the  peak  signal.  Classification  and  Type  were  detennined  from  the  direct  match  metric  or  the 
coherence  ratio  p,  as  applicable.  Easting,  Northing,  Depth,  Azimuth  and  Dip  all  came  from  the 
dipole  inversion  results. 


For  the  former  Camp  SLO  demonstration,  the  data  products  were  summarized  in  the  prioritized 
dig  list,  a  small  section  of  which  is  reproduced  in  Table  6-1  as  an  example.  The  ‘decision 
statistic’  refers  to  the  direct  match  metric  described  in  previous  sections.  The  ‘Rank’  column 
lists  the  anomalies  from  1  to  N  were  1  indicates  most  likely  non-TOI  and  N  indicates  high 
confidence  TOI.  The  ‘Category’  column  lists  an  integer  key  of  1  for  likely  clutter,  2  for  Can’t 
Decide,  3  for  likely  munitions,  or  4  for  Can’t  Analyze.  The  ‘Overlap’  column  lists  nearby 
anomalies  (if  any)  that  spatially  overlap.  The  ‘Type’  column  was  used  to  list  the  munitions  type. 
The  Easting,  Northing,  Depth,  Azimuth,  and  Dip  columns  reported  inversion  results.  The 
‘Items/ Anomaly’  column  reported  an  estimate  of  the  number  of  sources  that  contributed  to  the 
observed  anomaly. 

Table  6-1  -  Partial  prioritized  dig  list  for  the  former  Camp  SLO  TEMTADS  demonstration. 


MASTER  ID 

DECISION  STATISTIC 

Rank 

Category 

Overlaps 

Type 

Easting  (m) 

Northing  (m) 

Depth  (m) 

Azimuth 

Dip 

Items/ Anomaly 

441 

0.1107 

1047 

2 

0 

705280.41 

3913908.62 

0.559 

104.94 

32.94 

1 

538 

0.1088 

1048 

2 

0 

705301.47 

3913778.32 

-0.083 

169.97 

30.3 

1 

783 

0.1075 

1049 

3 

41 

60mm 

705343.03 

3913787.55 

0.503 

63.8 

39.68 

3 

825 

0.1072 

1050 

3 

0 

4.2in 

705349.68 

3913791.99 

0.652 

186.01 

9.29 

3 
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7.0  PERFORMANCE  ASSESSMENT 


The  TEMTADS  array  was  constructed  in  2007,  field  tested  at  the  APG  Standardized  UXO  Test 
Site  in  June  2008  [15],  and  participated  in  the  ESTCP  UXO  Classification  Study  at  the  former 
Camp  SLO  site  in  June  2009  [19]. 

Due  to  the  nature  of  the  demonstrations  at  each  site  and  the  employed  scoring  methods,  separate 
Performance  Objectives  are  provided  for  each  demonstration  in  the  following  subsections. 

7.1  APG  STANDARDIZED  UXO  TEST  SITE 

For  the  APG  demonstration,  a  magnetometer  data  set  collected  the  previous  month  at  the  newly 
reconfigured  test  site  was  used  for  anomaly  detection.  Approximately  200  cells  in  the  Blind  Grid 
and  700  anomalies  in  the  Indirect  Fire  Area  were  interrogated  with  the  TEMTADS,  averaging 
200  anomalies/day  on  the  Indirect  Fire  Area.  After  processing,  ranked  dig  lists  were  generated 
and  submitted  to  ATC  for  scoring. 

The  performance  objectives  for  this  demonstration  were  summarized  in  Table  3-1,  and  are 
repeated  here  as  Table  7-1.  The  results  for  each  criterion  are  then  discussed  in  the  following 
sections.  The  ATC  scoring  report  for  the  demonstration,  Reference  15,  provides  the  source 
material  for  evaluating  several  of  the  perfonnance  objectives. 
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Table  7-1  -  Performance  Results  for  the  APG  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Success? 

Quantitative  Objectives 

Reduction  of 

False  Alarms 

Number  of  false 
alarms 
eliminated  at 
demonstrator 
operating  point. 

•  Prioritized  dig 
list 

•  Scoring  report 
from  APG 

Reduction  of 
false  alarms  by  > 
50%  with  95% 

correct 

identification  of 
munitions 

Yes 

Location 

Accuracy 

Average  error 
and  standard 
deviation  in  both 
axes  for 
interrogated 
items 

•  Estimated 
location  from 
analysis 

•  Scoring  report 
from  APG 

AN  and  AE<  10 

cm 

ctN  and  ctE  <  1 5 

cm 

Yes 

Production  Rate 

Number  of 
targets 

interrogated  each 
day 

•  Log  of  field  work 

75  targets  per 
day 

Yes 

Analysis  Time 

Average  time 
required  for 
inversion  and 
classification 

•  Log  of  analysis 
work 

1 5  min  per  target 

Yes 

Qualitative  Objectives 

Ease  of  Use 

•  Feedback  from 
operator  on  ease 
of  use 

Operator  comes 
to  work  smiling 

Yes 

7.1.1  OBJECTIVE:  REDUCTION  OF  FALSE  ALARMS 

This  is  the  primary  measure  of  the  effectiveness  of  this  technology.  By  collecting  high-quality, 
precisely-located  data,  we  expect  to  be  able  to  discriminate  munitions  from  scrap  and  frag  with 
high  efficiency. 

7. 1.1.1. Metric 

At  a  seeded  test  site  such  as  the  APG  standardized  test  site,  the  metric  for  false  alarm  elimination 
is  straightforward.  We  prepared  a  ranked  dig  list  for  the  targets  we  interrogated  with  a  dig/no¬ 
dig  threshold  indicated  and  ATC  personnel  use  their  automated  scoring  algorithms  to  assess  our 
results. 
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7.1.1.2.Data  Requirements 


The  identification  of  most  of  the  items  in  the  test  field  is  known  to  the  test  site  operators.  Our 
ranked  dig  list  is  the  input  for  this  standard  and  ATC’s  standard  scoring  is  the  output. 

7.1. 1.3. Success  Criteria 

The  objective  will  be  considered  to  be  met  if  more  than  50%  of  the  non-munitions  items  can  be 
labeled  as  no-dig  while  retaining  95%  of  the  munitions  items  on  the  dig  list. 

7.1.1.4.Results 

This  Objective  was  successfully  met.  The  TEMTADS  surveyed  anomalies  detected  by  the 
MTADS  magnetometer  system  in  the  Blind  Grid  and  Indirect  Fire  Areas.  For  the  Blind  Grid 
Test  Area,  the  discrimination  stage  results  are  summarized  in  Table  7-2  and  Table  7-3  (subsets  of 
Table  6a  of  Reference  15),  broken  out  by  munitions  type  and  emplacement  depth.  For  the 
Indirect  Fire  Test  Area,  the  discrimination  stage  results  are  summarized  in  Table  7-4  and  Table 
7-5  (subsets  of  Table  6c  of  Reference  15),  broken  out  by  munitions  type  and  emplacement  depth. 
The  Discrimination  Stage  Probability  of  Detection  Pddlsc  is  defined  as  the  number  of  correctly 
identified  munitions  divided  by  the  number  of  emplaced  munitions,  and  the  corresponding 
Probability  of  False  Positive  Pfpdlsc  is  the  number  of  clutter  items  incorrectly  identified  as 
munitions  divided  by  the  number  of  emplaced  clutter  items. 

Table  7-2  -  TEMTADS  Blind  Grid  Test  Area  Pddisc  Results 


D  disc 

rd 

All  Types 

105- mm 

81/60mm 

37/25-mm 

Munitions 

Scores 

0.97 

0.93 

0.97 

1.00 

0  to  4D 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

8D  to  12D 

0.67 

0.67 

0.00 

1.00 

Table  7-3  -  TEMTADS  Blind  Grid  Test  Area  Pfpdisc  Results 


to  disc 

Efp 

All 

Masses 

0  to  0.25 
kg 

>0.25  to  1 
kg 

>1  to  10 
kg 

All  Depths 

0.01 

0.02 

0.00 

0.00 

0  to  0.15m 

0.01 

0.02 

0.00 

0.00 

0.15  to 

0.3m 

0.00 

0.00 

0.00 

0.00 

0.3  to 

0.6m 

N/A 

N/A 

N/A 

N/A 
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Table  7-4  -  TEMTADS  Indirect  Fire  Test  Area  Pddisc  Results 


n  disc 

"d 

All  Types 

105- mm 

81/60mm 

3 7/2 5- mm 

Munitions 

Scores 

0.92 

0.93 

0.93 

0.91 

By 

Density 

High 

0.88 

0.92 

0.91 

0.80 

Medium 

0.94 

0.97 

0.89 

0.97 

Low 

0.94 

0.90 

0.97 

0.94 

By  Depth 

0  to  4D 

0.96 

0.94 

0.97 

0.97 

4D  to  8D 

0.92 

0.94 

0.92 

0.86 

8D  to  12D 

0.72 

0.75 

0.78 

0.67 

Table  7-5  -  TEMTADS  Indirect  Fire  Test  Area  Pfpdisc  Results 


D  disc 

Rfp 

All 

Masses 

0  to  0.25 
kg 

>0.25  to  1 
kg 

>1  to  10 
kg 

All  Depths 

0.04 

0.03 

0.02 

0.11 

0  to  0.15m 

0.04 

0.04 

0.02 

0.13 

0.15  to 

0.3m 

0.04 

0.00 

0.06 

0.06 

0.3  to 

0.6m 

0.08 

0.00 

0.00 

0.20 

Discrimination  Efficiency  (E)  and  False  Positive  Rejection  Rate  (R|P)  measure  the  effectiveness 
of  the  discrimination  stage  processing.  The  goal  of  discrimination  is  to  retain  the  greatest 
number  of  munitions  detections  from  the  anomaly  list,  while  rejecting  the  maximum  number  of 
anomalies  arising  from  non-munitions  items.  Efficiency  measures  the  fraction  of  detected 
munitions  retained  after  discrimination,  while  the  rejection  rate  measures  the  fraction  of  false 
alarms  rejected.  The  measures  are  defined  relative  to  the  number  of  munitions  items  or  the 
number  of  clutter  items  that  were  actually  detected  by  the  sensor.  The  results  for  the  Blind  Grid 
and  Indirect  Fire  Test  Areas  are  summarized  in  Table  7-6  and  Table  7-7,  from  Tables  7a  and  7c 
of  Reference  15.  Perfonnance  levels  are  shown  at  two  specific  operating  points  on  the  ROC 
curve:  one  at  the  point  where  no  decrease  in  Pd  is  incurred  and  the  other  at  the  operator-selected 
operating  point  or  threshold.  In  the  Blind  Grid,  98%  of  the  emplaced  munitions  items  were 
detected.  Of  these,  99%  (97%  of  the  emplaced  munitions)  were  correctly  classified  at  our 
selected  operating  point,  with  a  corresponding  false  positive  rejection  rate  of  99%.  Moving  out 
the  ROC  to  the  point  where  100%  of  the  detected  munitions  items  are  correctly  classified 
reduces  the  false  positive  rejection  rate  to  95%.  In  the  Indirect  Fire  Area,  94%  of  the  emplaced 
munitions  were  detected,  and  92%  of  the  emplaced  munitions  were  correctly  classified,  resulting 
in  a  discrimination  efficiency  of  98%.  The  corresponding  false  positive  rejection  rate  was  92%. 
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Table  7-6  -  TEMTADS  Blind  Grid  Test  Area  Efficiency  and  Rejection  Rates 


Efficiency 

(E) 

False 

Positive 

Rejection 

Rate 

At  Operating 

Point 

0.99 

0.99 

With  No  Loss  of 
Pd 

1.00 

0.95 

Table  7-7  -  TEMTADS  Indirect  Fire  Test  Area  Efficiency  and  Rejection  Rates 


Efficiency 

(E) 

False 

Positive 

Rejection 

Rate 

At  Operating 

Point 

0.98 

0.92 

With  No  Loss  of 
Pd 

1.00 

0.58 

7.1.2  OBJECTIVE:  LOCATION  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  location  of  the  targets  marked  to  be  dug.  Large  location  errors  lead  to  confusion 
among  the  UXO  techs  assigned  to  the  remediation  costing  time  and  often  leading  to  removal  of  a 
small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

7.1.2.  l.Metric 

As  above,  the  metric  for  location  accuracy  is  straightforward  at  a  seeded  test  site  such  as  the 
APG  standardized  test  site.  We  provided  an  estimated  position  for  all  targets  we  interrogated 
and  ATC  personnel  used  their  automated  scoring  algorithms  to  assess  our  results. 

7.1.2.2. Data  Requirements 

The  location  of  most  of  the  items  in  the  test  field  is  known  to  the  test  site  operators.  Our  dig  list 
is  the  input  for  this  standard  and  ATC’s  standard  scoring  is  the  output. 

7.I.2.3. Success  Criteria 

The  objective  will  be  considered  to  be  met  if  the  average  position  error  was  less  than  10  cm  in 
both  dimensions  (low  bias)  and  the  standard  deviation  of  each  dimension  was  less  than  15  cm 
(accurate  location). 
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7.1.2.4.Results 


This  Objective  was  successfully  met.  The  location  accuracy  of  fit  parameters  generated  from  the 
TEMTADS  array  data  are  given  in  Table  7-8  and  Table  7-9,  taken  from  Tables  9a  and  9c  of 
Reference  15.  Horizontal  errors  are  not  calculated  for  the  Blind  Grid. 

Table  7-8  -  TEMTADS  Blind  Grid  Test  Area  Location  Error  and  Standard  Deviation 


Mean 

(m) 

Standard 

Deviation 

(m) 

Northing 

N/A 

N/A 

Easting 

N/A 

N/A 

Depth 

0.02 

0.04 

Table  7-9  -  TEMTADS  Indirect  Fire  Test  Area  Location  Error  and  Standard  Deviation 


Mean 

(m) 

Standard 

Deviation 

(m) 

Northing 

0.01 

0.05 

Easting 

0.01 

0.05 

Depth 

0.00 

0.06 

7.1.3  OBJECTIVE:  PRODUCTION  RATE 

Even  if  the  performance  of  the  technology  on  the  two  metrics  above  is  satisfactory,  there  is  an 
economic  metric  to  consider.  There  is  a  known  cost  of  remediating  a  suspected  munitions  item. 
If  the  cost  to  interrogate  a  target  is  greater  than  this  cost,  the  technology  will  be  useful  only  at 
sites  with  special  conditions  or  target  values.  Note,  however,  that  in  its  ultimate  implementation 
this  technology  will  result  in  reacquisition,  cued  interrogation,  and  target  flagging  in  one  visit  to 
the  site. 

7.1.3.  l.Metric 

The  number  of  targets  interrogated  per  day  is  the  metric  for  this  objective.  Combined  with  the 
daily  operating  cost  of  the  technology  this  gives  the  per-item  cost. 

7.1.3.2.Data  Requirements 

Survey  productivity  was  detennined  from  a  review  of  the  ATC  demonstration  field  logs. 
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7.I.3.3. Success  Criteria 


For  this  first  demonstration,  the  objective  will  be  considered  to  be  met  if  at  least  75  targets  were 
interrogated  each  survey  day. 

7.1.3.4. Results 

This  Objective  was  successfully  met.  Data  collection  times  are  taken  from  Table  5  of  Reference 
15.  For  the  purposes  of  this  discussion,  a  typical  work  day  is  assumed  to  be  7  hours  of  active 
data  collection  and  1  hour  of  daily  setup  and  tear  down.  214  anomalies  were  investigated  on  the 
Blind  Grid  over  the  course  of  10  hours  and  3  minutes,  or  1.43  work  days,  or  on  average  149 
anomalies  /  work  day.  694  anomalies  were  investigated  in  the  Indirect  Fire  Area  over  the  course 
of  32  hours  and  30  minutes,  or  on  average  150  anomalies  /  work  day.  In  fact,  through  the 
generosity  of  the  ATC  staff,  we  were  able  to  work  for  ten  or  more  hours  on  several  days  and 
achieved  production  rates  of  >  220  anomalies  /  day  for  three  days  at  the  end  of  the  first  week  of 
the  demonstration. 

7.1.4  OBJECTIVE:  ANALYSIS  TIME 

The  ultimate  implementation  of  this  technology  will  involve  on-the-fly  analysis  and 
classification.  The  time  for  this  will  be  limited  to  the  driving  time  to  the  next  anomaly  on  the 
list.  We  tracked  the  near-real-time  analysis  time  in  this  demonstration. 

7. 1.4.1.  Metric 

The  time  required  for  inversion  and  classification  per  anomaly  is  the  metric  for  this  objective 

7.1.4.2. Data  Requirements 

Analysis  time  will  be  determined  from  a  review  of  the  data  analysis  logs. 

7.I.4.3. Success  Criteria 

For  this  first  demonstration,  the  objective  will  be  considered  to  be  met  if  the  average  inversion 
and  classification  time  was  less  than  15  min. 

7.1.4.4. Results 

This  Objective  was  successfully  met.  The  average  inversion  time  per  target  was  approximately 

2.5  minutes  on  our  field  laptop  computer.  Including  this,  the  average  analysis  time  amounted  to 

12.5  minutes  per  anomaly.  Since  this  was  the  first  extensive  test  of  the  system  in  field  mode,  we 
took  the  opportunity  to  consider  various  discrimination  and  classification  methods,  some  of 
which  proved  unfruitful.  As  a  result  of  lessons  learned  from  this  undertaking,  we  expect  the 
average  analysis  time  for  future  field  runs  to  be  less  than  that  obtained  here. 
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7.1.5  OBJECTIVE:  EASE  OF  USE 


This  qualitative  objective  is  intended  as  a  measure  of  the  long-tenn  usability  of  the  technology. 
If  the  operator  does  not  report  that  the  technology  is  easy  to  use,  shortcuts  that  can  compromise 
the  efficiency  of  the  technology  will  begin  to  creep  into  daily  operations. 

7.1.5.1. Data  Requirements 

This  objective  was  evaluated  based  on  operator  feedback. 

7.1.5.2. Results 

This  Objective  was  successfully  met.  Based  on  vehicle  operator  feedback,  there  were  no 
significant  limitations  to  the  efficient  use  of  the  system  in  the  field.  Several  suggestions  were 
made  for  additional  improvements  to  the  navigation  and  data  collection  software.  They  have 
been  subsequently  incorporated.  Anomalies  for  future  demonstrations  will  be  ordered  along  lm 
swaths  for  maximum  data  collection  efficiency. 

7.2  FORMER  CAMP  SAN  LUIS  OBISPO 

The  performance  objectives  for  the  demonstration  were  presented  in  Table  3-2,  and  are  repeated 
here  as  Table  7-10. 

Prior  to  our  demonstration  run  at  the  former  Camp  SLO,  we  obtained  measurements  of  each  of 
the  four  munitions  types.  Using  the  derived  polarizabilities  from  these  measurements  and  the 
forward  model  of  our  TEMTADS  code,  response  curves  were  generated  for  each  munition. 
These  curves  plot  the  minimum  expected  peak  signal  (the  peak  signal  when  the  target  is  oriented 
horizontally)  and  the  maximum  expected  peak  (the  peak  signal  when  the  target  is  oriented 
horizontally)  from  each  target  as  a  function  of  distance  below  sensor  or  depth  below  ground. 
The  peak  signal  is  that  obtained  from  the  monostatic  term  of  the  center  element  of  the  array  at 
the  first  time  gate. 

A  calibration  strip,  comprised  of  two  samples  of  each  of  the  four  targets  of  interest  plus  two 
shotputs  (Table  7-11),  was  emplaced  on  site  as  a  means  of  verifying  proper  system  operation  on 
a  daily  basis.  The  strip  was  surveyed  twice  daily,  once  at  the  beginning  of  the  day,  and  once  at 
the  end.  The  sole  exception  to  this  procedure  occurred  on  our  third  day,  when  brake  problems 
with  the  vehicle  forced  an  early  cessation  of  activities. 
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Table  7-10  -  Performance  Objectives  for  the  former  Camp  SLO  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Success? 

Site  Coverage 

Fraction  of 
assigned  anomalies 
interrogated 

Survey  results 

1 00%  as  allowed  for 
by  topography  / 
vegetation 

Yes 

Calibration 

Strip  Results 

System  response 
consistently 
matches  physics- 
based  model 

•  System  response 
curves 

•  Daily 

calibration  strip 
data 

•  <  15%  rms  variation 
in  amplitude 

•  Down-track  location 
±  25cm 

•  All  response  values 
fall  within  bounding 
curves 

No 

Location 

Accuracy 

Average  error  and 
standard  deviation 
in  both  axes  for 
interrogated  items 

•  Estimated 
location  from 
analyses 

•  Ground  truth 
from  validation 
effort 

AN  and  AE  <  5  cm 
ctN  and  ctE  <  1 0  cm 

No 

Depth 

Accuracy 

Standard  deviation 
in  depth  for 
interrogated  items 

•  Estimated 
location  from 
analyses 

•  Ground  truth 
from  validation 
effort 

ADepth  <  5  cm 
crDepth  <10  cm 

No 

Production 

Rate 

Number  of 
anomalies 
investigated  each 
day 

•  Survey  results 

•  Log  of  field 
work 

125  anomalies/day 

Yes 

Data 

Throughput 

Throughput  of  data 
QC  process 

Log  of  analysis 
work 

All  data  QC’ed  on  site 
and  at  pace  with  survey 

Yes 

Reliability  and 
Robustness 

General 

Observations 

Team  feedback 

Field  team  comes  to 
work  smiling 

Yes 
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Table  7-1 1  -  Details  of  former  Camp  SLO  Calibration  Strip 


Item 

ID 

Description 

Easting  (m) 

Northing  (m) 

Depth  (m) 

Inclination 

Azimuth 
(°cw  from  N) 

T-001 

shotput 

705,417.00 

3,913,682.00 

0.25 

N/A 

N/A 

T-002 

81mm 

705,420.92 

3,913,687.63 

0.30 

Vertical  Down 

0 

T-003 

81mm 

705,424.10 

3,913,692.95 

0.30 

Horizontal 

120 

T-004 

60mm 

705,427.53 

3,913,698.54 

0.30 

Vertical  Down 

0 

T-005 

60mm 

705,430.85 

3,913,704.10 

0.30 

Horizontal 

120 

T-006 

4.2”  mortar 

705,434.54 

3,913,709.44 

0.30 

Vertical  Down 

0 

T-007 

4.2”  mortar 

705,437.99 

3,913,715.04 

0.30 

Horizontal 

120 

T-008 

2.36”  rocket 

705,441.46 

3,913,720.24 

0.30 

Vertical  Down 

0 

T-009 

2.36”  rocket 

705,445.00 

3,913,725.91 

0.30 

Horizontal 

120 

T-010 

shotput 

705,448.50 

3,913,731.50 

0.35 

N/A 

N/A 

7.2.1  OBJECTIVE:  SITE  COVERAGE 

A  list  of  previously  identified  anomalies  was  provided  by  the  Program  Office. 

7.2.1.  l.Metric 

Site  coverage  is  defined  as  the  fraction  of  the  assigned  anomalies  surveyed  by  the  TEMTADS. 

7.2.1.2.Data  Requirements 

The  collected  data  were  compared  to  the  original  anomaly  list.  All  interferences  were  noted  in 
the  field  log  book. 

7.2. 1.3. Success  Criteria 

The  objective  was  considered  met  if  100%  of  the  assigned  anomalies  are  surveyed  with  the 
exception  of  areas  that  can  not  be  surveyed  due  to  topology  /  vegetation  interferences. 

7.2.1.4.Results 

This  objective  was  successfully  met.  Of  the  list  provided  by  the  Program  Office,  all  but  5  were 
measured.  Failure  to  measure  these  5  anomalies  was  due  to  the  presence  of  rocks  which 
prevented  the  operator  from  positioning  the  TEMTADS  over  the  target. 

7.2.2  OBJECTIVE:  CALIBRATION  STRIP  RESULTS 

This  objective  supports  that  each  sensor  system  is  in  good  working  order  and  collecting 
physically  valid  data  each  day.  The  calibration  strip  was  surveyed  twice  daily.  The  peak 
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positive  response  of  each  emplaced  item  from  each  run  was  compared  to  the  physics-based 
response  curves  generated  prior  to  data  collection  on  site  using  each  item  of  interest. 

7.2.2. 1. Metric 

The  reproducibility  of  the  measured  response  of  each  sensor  system  to  the  items  of  interest  and 
the  comparison  of  the  response  to  the  response  predicted  by  the  physics-based  model  defines  this 
metric. 

7.2.2.2. Data  Requirements 

Response  curves  for  each  sensor  /  item  of  interest  pair  are  used  to  document  what  the  physics- 
based  response  of  the  system  to  the  item  should  be.  The  tabulated  peak  response  values  from 
each  survey  of  the  Calibration  Strip  demonstrates  the  reproducibility  and  validity  of  the  sensor 
readings. 

1. 2.2.3. Success  Criteria 

The  objective  is  considered  met  if  all  measured  responses  fall  within  the  range  of  physically 
possible  values  based  on  the  appropriate  response  curve.  Additionally,  the  RMS  variation  in 
responses  should  be  less  than  15%  of  the  measured  response  and  the  down-track  location  of  the 
anomaly  should  be  within  25  cm  of  the  corresponding  seeded  item’s  true  location. 

7.2.2.4.Results 

This  objective  was  not  successfully  met  in  full.  The  measured  peak  signals  for  all  of  the 
emplaced  items  are  shown  in  Figure  7-1  through  Figure  7-5.  The  maximum  (red)  and  minimum 
(blue)  response  curves  are  plotted  for  all  objects  except  the  shotput,  which  has  only  one  curve 
due  to  symmetry.  The  two  curves  for  the  81 -mm  and  4.2-in  mortar  are  nearly  equal.  For  self- 
consistency,  we  have  plotted  the  measurements  at  the  mean  inverted  depth,  rather  than  the 
reported  depths.  The  measured  values  generally  fit  well  within  the  bounding  curves,  with  the 
4.2-in  mortar  and  shotput  results  the  poorest,  with  a  tendency  to  underestimate  the  peak  value. 

Careful  examination  of  the  data  shows  that  this  variation  is  the  result  of  the  shot-to-shot 
precision  with  which  the  array  can  be  positioned  in  exactly  the  same  spot  each  time.  Because  the 
response  curves  are  generated  assuming  the  target  is  directly  below  the  sensor,  any  offset  in  the 
sensor  position  will  result  in  the  derived  peak  signal  being  smaller  than  that  predicted  by  the 
curve,  as  is  observed.  For  future  demonstrations,  the  metric  of  polarizability  amplitude  is 
recommended  as  these  values  are  invariant  to  array  position. 

Table  7-12  shows  the  mean  and  standard  deviation  in  the  peak  measured  signal  for  all  of  the 
emplaced  Calibration  Strip  items.  Of  the  10  items,  three  give  RMS  variations  above  our  stated 
goal  of  15%,  with  only  T-008  being  significantly  above.  This  target  consistently  inverted  to  a 
depth  of  18.5  cm,  and  thus  showed  the  largest  spatial  variation  in  signal.  This,  coupled  with  the 
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array  not  always  being  properly  centered  on  the  target,  explains  the  larger  variation  for  this 
object. 

Table  7-12  -  Peak  Signals  for  former  Camp  SLO  Calibration  Strip  Emplaced  Items 


Item 

ID 

Description 

Depth 

(m) 

Mean  Signal 
(mV/Amp) 

Std  Dev. 
(mV/Amp,lo) 

Variation 

(%) 

T-001 

shotput 

0.25 

27.44 

4.18 

15.23 

T-002 

81mm 

0.30 

15.46 

1.39 

8.99 

T-003 

81mm 

0.30 

10.96 

0.91 

8.30 

T-004 

60mm 

0.30 

8.74 

0.89 

10.18 

T-005 

60mm 

0.30 

6.73 

1.13 

16.79 

T-006 

4.2”  mortar 

0.30 

52.38 

5.82 

11.11 

T-007 

4.2”  mortar 

0.30 

41.75 

5.25 

12.57 

T-008 

2.36”  rocket 

0.30 

32.70 

7.91 

24.19 

T-009 

2.36”  rocket 

0.30 

4.04 

0.32 

7.92 

T-010 

shotput 

0.35 

11.82 

1.52 

12.86 

Figure  7-1  -  Peak  signals  compared  with  response  curve  for  a  2.36-in  rocket. 
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Peak  Signal(mV/A) 


Figure  7-2  -  Peak  signals  compared  with  response  curve  for  a  4.2-in  mortar. 


Figure  7-3  -  Peak  signals  compared  with  response  curve  for  a  60-mm  mortar. 
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Figure  7-4  -  Peak  signals  compared  with  response  curve  for  an  81 -mm  mortar. 
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Figure  7-5  -  Peak  signals  compared  with  response  curve  for  a  4-in  diameter  shotput. 

The  variability  and  accuracy  of  the  positional  fit  parameters  for  the  Calibration  Strip  emplaced 
items  were  determined  by  comparing  the  inverted  Northing  and  Easting  values  with  reported 
values.  These  numbers  are  shown  in  Table  7-13.  We  give  the  mean  vector  offset  dR  between 
the  inverted  and  reported  position  as  well  as  the  Easting  (dx)  and  Northing  (dy)  components. 
The  dx  and  dy  values  are  computed  using  the  inverted  positions  minus  the  reported  ones. 
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Two  points  are  clear  from  the  values  in  Table  7-13.  First,  the  inversion  process  is  very  robust, 
with  no  standard  deviations  larger  than  2cm.  Second,  there  are  a  few  large  discrepancies 
between  our  inverted  positions  and  the  reported  ones.  All  offsets  are  below  our  target  value  of 
25cm,  with  the  exception  of  T-002.  It  is  possible  that  some  items  have  drifted  or  settled  since 
their  original  emplacement. 

Table  7-13  -  Position  Accuracy  and  Variability  for  former  Camp  SLO  Calibration  Strip  Emplaced  Items 


Item 

ID 

Description 

Depth 

(m) 

Mean  dR 
(m) 

Std  Dev 
dR 

(m,  lo) 

Mean 

dx 

(m) 

Std  Dev 

dx 

(m) 

Mean 

dy 

(m) 

Std  Dev 

dy 

(m) 

T-001 

shotput 

0.25 

0.209 

0.007 

0.167 

0.013 

0.124 

0.011 

T-002 

81mm 

0.30 

0.413 

0.008 

-0.372 

0.008 

-0.178 

0.010 

T-003 

81mm 

0.30 

0.058 

0.014 

0.028 

0.010 

-0.049 

0.020 

T-004 

60mm 

0.30 

0.026 

0.013 

0.013 

0.010 

0.015 

0.019 

T-005 

60mm 

0.30 

0.038 

0.012 

-0.003 

0.011 

0.035 

0.017 

T-006 

4.2”  mortar 

0.30 

0.098 

0.008 

0.082 

0.009 

0.054 

0.009 

T-007 

4.2”  mortar 

0.30 

0.063 

0.008 

0.061 

0.009 

0.013 

0.010 

T-008 

2.36”  rocket 

0.30 

0.035 

0.010 

-0.027 

0.011 

0.018 

0.011 

T-009 

2.36”  rocket 

0.30 

0.112 

0.016 

0.051 

0.010 

-0.099 

0.018 

T-010 

shotput 

0.35 

0.166 

0.006 

0.133 

0.008 

-0.098 

0.014 

7.2.3  OBJECTIVE:  LOCATION  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  the  predicted  location  of  the  targets  marked  to  be  dug.  Large  location  errors  lead  to  confusion 
among  the  UXO  technicians  assigned  to  the  remediation,  costing  time  and  often  leading  to 
removal  of  a  small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

7.2.3.  l.Metric 

The  average  error  and  standard  deviation  in  both  horizontal  axes  will  be  computed  for  the  items 
selected  for  excavation  during  the  validation  phase  of  the  study. 

7.2.3.2.Data  Requirements 

The  anomaly  fit  parameters  and  the  ground  truth  for  the  excavated  items  are  required  to 
determine  the  performance  of  the  fitting  routines  in  terms  of  the  location  accuracy. 

7.2.3.3. Success  Criteria 

This  objective  is  considered  as  met  if  the  average  error  in  position  for  both  Easting  and  Northing 
quantities  is  less  than  5  cm  and  the  standard  deviation  for  both  is  less  than  10  cm. 
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7.2.3.4.Results 


This  objective  was  not  successfully  met  in  full.  The  average  Northing  position  error  for  all 
measured  data  was  1.5cm,  while  the  average  Easting  position  error  was  3cm.  The  standard 
deviations,  however,  were  larger  than  desired,  with  both  values  about  25cm.  Excluding  those 
anomalies  for  which  multiple  targets  were  found  produces  negligible  improvement.  We  suspect 
the  higher  values  are  due  to  the  large  number  of  small,  low  SNR  clutter  items,  which  result  in 
greater  uncertainty  in  both  the  measured  and  fitted  values.  Indeed,  if  we  restrict  ourselves  to 
those  anomalies  which  we  classified  as  ‘likely  UXO,’  the  Northing  and  Easting  standard 
deviations  drop  to  7cm  and  5cm,  respectively. 

7.2.4  OBJECTIVE:  DEPTH  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  the  predicted  depth  of  the  targets  marked  to  be  dug.  Large  depth  errors  lead  to  confusion 
among  the  UXO  technicians  assigned  to  the  remediation,  costing  time  and  often  leading  to 
removal  of  a  small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

7.2.4.1. Metric 

The  standard  deviation  of  the  predicted  depths  with  respect  to  the  ground  truth  will  be  computed 
for  the  items  which  are  selected  for  excavation  during  the  validation  phase  of  the  study. 

7.2.4.2. Data  Requirements 

The  anomaly  fit  parameters  and  the  ground  truth  for  the  excavated  items  are  required  to 
determine  the  performance  of  the  fitting  routines  in  terms  of  the  predicted  depth  accuracy. 

7 .2.4.3. Success  Criteria 

This  objective  is  considered  as  met  if  the  average  error  in  depth  is  less  than  5  cm  and  the 
standard  deviation  is  less  than  10  cm. 

7.2.4.4.Results 

This  objective  was  not  successfully  met  in  full.  The  average  depth  error  for  all  measured  data 
was  2.5cm.  The  standard  deviation  was  a  bit  larger  than  desired,  with  a  value  of  14cm.  As  we 
did  for  the  Northing  and  Easting,  we  tried  excluding  those  anomalies  for  which  multiple  targets 
were  found.  This  lowered  the  value  to  11cm.  Restricting  ourselves  to  those  anomalies  that  we 
classified  as  ‘likely  UXO,’  the  standard  deviation  is  lowered  to  a  value  of  7cm.  This  approach 
attempts  to  exclude  the  numerous,  small,  low  SNR  clutter  items  from  the  analysis. 
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7.2.5  OBJECTIVE:  PRODUCTION  RATE 


This  objective  considers  a  major  cost  driver  for  the  collection  of  high-density,  high-quality 
geophysical  data,  the  production  rate.  The  faster  high  quality  data  can  be  collected,  the  higher 
the  financial  return  on  the  data  collection  effort. 

7.2.5.1. Metric 

The  number  of  anomalies  investigated  per  day  determines  the  production  rate  for  a  cued  survey 
system. 

7.2.5.2. Data  Requirements 

The  metric  can  be  detennined  from  the  combination  of  the  field  logs  and  the  survey  results.  The 
field  logs  record  the  amount  of  time  per  day  spent  acquiring  the  data  and  the  survey  results 
determine  the  number  of  anomalies  investigated  in  that  time  period. 

7.2. 5.3. Success  Criteria 

This  objective  is  considered  met  if  average  production  rate  is  at  least  125  anomalies  /  day. 

7.2.5.4.Results 

This  objective  was  successfully  met.  A  total  of  1547  anomalies  (including  redo’s)  were  measured 
over  a  10-day  run  for  an  average  of  155  anomalies/day.  The  only  days  for  which  our  average  fell 
below  our  goal  of  125  anomalies/day  were  the  final  day,  due  to  finishing  up  all  targets,  and  2 
days  in  which  necessary  vehicle  repairs  shortened  our  workday. 

7.2.6  OBJECTIVE:  DATA  THROUGHPUT 

The  collection  of  a  complete,  high-quality  data  set  with  the  sensor  platform  is  critical  to  the 
downstream  success  of  the  UXO  Classification  Study.  This  objective  considers  one  of  the  key 
data  quality  issues,  the  ability  of  the  data  analysis  workflow  to  support  the  data  collection  effort 
in  a  timely  fashion.  To  maximize  the  efficient  collection  of  high  quality  data,  a  series  of 
MTADS  standard  data  quality  check  are  conducted  during  and  immediately  after  data  collection 
on  site.  Data  which  pass  the  QC  screen  are  then  processed  into  archival  data  stores.  Individual 
anomaly  analyses  were  then  conducted  on  those  archival  data  stores.  The  data  QC  / 
preprocessing  portion  of  the  workflow  needs  to  keep  pace  with  the  data  collection  effort  for  best 
performance. 

7.2.6.1.Metric 

The  throughput  of  the  data  quality  control  workflow  is  at  least  as  fast  the  data  collection  process, 
providing  real  time  feedback  to  the  data  collection  team  of  any  issues. 
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7.2.6.2.Data  Requirements 

The  data  analyst’s  log  books  will  provide  the  necessary  data  for  determining  the  success  of  this 
metric. 

7.2. 6.3. Success  Criteria 

This  objective  is  considered  met  if  all  collected  data  can  be  processed  through  the  data  quality 
control  portion  of  the  workflow  in  a  timely  fashion. 

7.2.6.4.Results 

This  objective  was  successfully  met.  Data  were  normally  downloaded  several  times  during  each 
workday,  and  quality  control  on  these  datasets  was  usually  completed  on  the  same  day.  Quality 
control  checks  successfully  caught  missed  anomalies,  a  small  number  of  corrupt  data  files,  and 
targets  which  needed  re-measuring. 

7.2.7  OBJECTIVE:  RELIABILITY  AND  ROBUSTNESS 

This  objective  represents  an  opportunity  for  all  parties  involved  in  the  data  collection  process, 
especially  the  vehicle  operator,  to  provide  feedback  on  areas  where  the  process  could  be 
improved. 

7.2.7.1. Data  Requirements 

This  decision  was  based  on  verbal  discussions,  communications,  and  opinions  of  the  entire  field 
team. 

7.2.7.2. Results 

This  objective  was  successfully  met.  Based  on  vehicle  operator  feedback,  there  were  no 
significant  limitations  to  the  efficient  use  of  the  system  in  the  field. 


8.0  COST  ASSESSMENT 

The  costs  for  a  3,000  anomaly  cued  survey  using  the  TEMTADS  are  detailed  in  Table  8-1.  The 
costs  associated  with  the  former  Camp  SLO  demonstration  were  used  as  the  basis  for  this 
assessment.  The  costs  reported  here  do  not  include  any  costs  associated  with  the  reconnaissance 
data  that  were  used  to  identify  and  create  the  cued  list  of  anomalies.  Implicit  in  our  assessment 
is  that  that  the  3,000  anomalies  are  bounded  within  an  area  of  roughly  25  acres,  or  120 
anomalies/acre.  Survey  time  would  increase  incrementally  as  the  anomaly  density  decreased  as 
transit  time  between  anomalies  increased.  All  costs  are  estimated  in  FY  2009  dollars. 
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8.1  COST  MODEL 


A  cost  model  was  constructed  from  the  costs  associated  with  the  TEMTADS  survey  of  the 
former  Camp  SLO  demonstration  site.  Cost  categories  were  mobilization,  logistics  (if  required), 
field  work,  demobilization,  data  analysis,  and  reporting.  Based  on  the  demonstrated  production 
rate  for  the  former  Camp  SLO  demonstration,  four  weeks  of  field  work  would  be  required. 
Based  on  the  production  rate  for  the  APG  demonstration,  the  SLO  production  rate  is  a 
conservative  lower  bound.  This  model  does  not  include  the  cost  of  anomaly  classification  and 
producing  a  prioritized  dig  list. 
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Table  8-1  -  Summary  of  Costs  for  a  25-acre,  3000  anomaly  TEMTADS  survey. 


Cost  Category 
Mobilization  Costs 


Logistics 
(if  required) 


Operating  Costs 
(4-week  Survey) 


Analysis  &  Reporting 


Demobilization  Costs 


Total  Cost 


Sub-Category  Cost 


Preliminary  Site  Visit 

$6,500 

Test  Plan  Preparation 

$10,250 

Equipment  Prep  and  Packing 

$9,750 

53’  Trailer  Transportation 

$5,800 

Analysts  Set-up 

$10,000 

Outbound  Travel  for  3  Personnel 

$2,250 

Establish  GPS  Control  Points 

$2,000 

Delivery/Removal  of  Logistics  Items 

$2,000 

Rental  of  Logistics  Items  /  Fuel 

$6,000 

Materials 

$6,500 

Supervisor 

$19,000 

On-site  Analyst 

$31,800 

Vehicle  Operator 

$12,750 

Field  Technician  (2  each) 

$17,250 

Per  Diem  (3  Personnel  x  4  weeks  x  $1250) 

$15,000 

Rental  Vehicles 

$8,650 

System  Maintenance 

$13,000 

Sensor  Repair 

$21,600 

Data  Reduction  to  Anomaly  List 

$28,000 

Demonstration  Data  Report 

$16,750 

Equipment  Unpacking 

$3,500 

53’  Trailer  Transportation 

$5,800 

Inbound  Travel  for  3  Personnel 

$2,250 

Sub-Total 

$44,550 


$16,500 


$139,050 


$44,750 


$11,550 


$256,400 
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8.2  COST  DRIVERS 


Two  factors  were  expected  to  be  strong  drivers  of  cost  for  this  technology  as  demonstrated.  The 
first  is  the  number  of  anomalies  which  can  be  surveyed  per  day.  Higher  productivity  in  data 
collection  equates  to  more  anomalies  investigated  for  a  given  period  of  time  in  the  field.  The 
time  required  for  analyzing  individual  anomalies  can  be  significantly  higher  than  for  other,  more 
traditional  methods  and  could  become  a  cost  driver  due  to  the  time  involvement.  The  thoughtful 
use  of  available  automation  techniques  for  individual  anomaly  analysis  with  operator  QC  support 
can  moderate  this  effect. 

8.3  COST  BENEFIT 

The  main  benefit  to  using  a  UXO  classification  process  is  cost-related.  The  ability  to  reduce  the 
number  of  non-hazardous  items  that  have  to  be  dug  or  dug  as  hazardous  directly  reduces  the  cost 
of  a  remediation  effort.  The  additional  information  for  anomaly  classification  provided  by  the 
TEMTADS  array  provides  additional  information  for  the  purposes  of  anomaly  classification.  If 
there  is  buy-in  from  the  stakeholders  to  use  these  techniques,  this  infonnation  can  be  used  to 
reduce  costs. 

9.0  IMPLEMENTATION  ISSUES 

Implementation  issues  for  this  system  /  technology  fall  into  two  categories:  operational  concerns 
and  data  quality  /  analysis  issues.  In  terms  of  operational  concerns,  the  TEMTADS  as 
implemented  is  a  large,  vehicle-towed  system  that  operates  best  in  large,  open  areas.  As  seen  at 
the  former  Camp  SLO  demonstration,  it  is  possible  to  operate  the  system  in  rocky  terrain  with 
grades  approaching  20%,  but  at  reduced  operating  capacity  and  increased  system  wear.  Smaller 
versions  of  the  system  are  currently  under  development  under  several  ESTCP  and  SERDP 
projects  to  address  these  concerns.  The  goal  is  to  design  and  field  units  more  amenable  to 
operation  in  more  confined  terrain  and  topology  and  smaller  tow  vehicles  /  man-portable  and 
handheld  operation.  In  terms  of  data  quality,  one  pays  a  small  penalty  in  signal  amplitude  to  use 
the  smaller  TEMTADS  sensor  coil  as  compared  to  other  systems  such  as  a  high-power  EM61 
Mkll.  The  dramatically  improved  electrical  perfonnance  of  these  new  sensors  helps 
counterbalance  this  issue,  particularly  in  the  ability  to  perform  better  averaging  (or  stacking). 
However,  one  needs  in  place  robust  data  QC  techniques  to  know  when  to  employ  these 
capabilities  in  an  efficient  manner. 
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Appendix  A:  Points  of  Contact 


POINT  OF 
CONTACT 

ORGANIZATION 

Phone 

Fax 

e-mail 

Role  in 
Project 

Dr.  Jeff 
Marqusee 

ESTCP  Program  Office 

901  North  Stuart  Street, 
Suite  303 

Arlington,  VA  22203 

703-696-2120  (V) 
703-696-2114  (F) 
jeffrey.marqusee@osd.mil 

Director, 

ESTCP 

Dr.  Anne 
Andrews 

ESTCP  Program  Office 

901  North  Stuart  Street, 
Suite  303 

Arlington,  VA  22203 

703-696-3826  (V) 
703-696-2114  (F) 
anne.andrews@osd.mil 

Deputy 

Director, 

ESTCP 

Dr.  Herb 

Nelson 

ESTCP  Program  Office 

901  North  Stuart  Street, 
Suite  303 

Arlington,  VA  22203 

703-696-3726  (V) 
703-696-2114  (F) 
202-215-4844  (C) 
herbert.nelson@osd.mil 

Program 
Manager,  MM 

Mr.  Peter 
Knowles 

HydroGeoLogic,  Inc. 

11107  Sunset  Hills  Road, 
Suite  400 

Reston,  VA  20190 

703-736-4511  (V) 
pknowles@hgl.com 

Program 
Manager’s 
Assistant,  MM 

Dr.  Dan 
Steinhurst 

Naval  Research  Lab 
Chemistry  Division 

Code  61 10 

Washington,  DC  20375 

202-767-3556  (V) 
202-404-8119  (F) 
703-850-5217  (C) 
dan.steinhurst@nrl.navy.mil 

Co-PI  and 

Data  Analyst 

Dr.  Tom  Bell 

SAIC 

200  12th  Street  South 
Arlington,  VA  22202 

703-414-3904  (V) 
703-413-0505  (F) 
301-712-7021  (C) 
thomas.h.bell@saic.com 

Co-PI  and 
Quality 

Control 

Officer 

Mr.  Rick  Fling 

Aberdeen  Test  Center 

410-278-2999  (V) 
301-992-9080  (C) 
rick.fling@us.army.mil 

Test  Site 
Manager 

Mr.  David 
Ragsdale 

California  Polytechnic 

State  University 

San  Luis  Obispo,  CA 

93407 

805-756-6662  (V) 
805-756-1602  (F) 
dragsdal@calpoly.edu 

Environmental 
Health  & 

Safety 

Manager  / 

Risk 

Management 
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